You can override the default parameters of a node inside a pipeline using the override attribute.
In the example below we're creating a pipeline that runs a preprocessing step and three training steps.
The three training nodes are overriding either the default input data, the parameters, or both.
When overriding parameters keep in mind that:
- the whole parameter section gets overridden, meaning you always need to specify all the parameters of that step. Even the ones that keep their default value.
- you should always specify the name, type, and default value when overriding a parameter
- step:
name: preprocess-dataset
image: python:3.9
command:
- pip install numpy valohai-utils
- python ./preprocess_dataset.py
inputs:
- name: dataset
default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz
- step:
name: train-model
image: tensorflow/tensorflow:2.6.0
command:
- pip install valohai-utils
- python ./train_model.py {parameters}
parameters:
- name: epochs
default: 5
type: integer
- name: learning_rate
default: 0.001
type: float
inputs:
- name: dataset
default: https://valohaidemo.blob.core.windows.net/mnist/preprocessed_mnist.npz
- pipeline:
name: Three-Trainings Pipeline
nodes:
- name: preprocess
type: execution
step: preprocess-dataset
- name: train1
type: execution
step: train-model
override:
inputs:
- name: dataset
- name: train2
type: execution
step: train-model
override:
inputs:
- name: dataset
parameters:
- name: epochs
default: 10
type: integer
- name: learning_rate
default: 0.001
type: float
- name: train3
type: execution
step: train-model
override:
inputs:
- name: dataset
parameters:
- name: epochs
default: 15
type: integer
- name: learning_rate
default: 0.001
type: float
edges:
- [preprocess.output.preprocessed_mnist.npz, train1.input.dataset]
- [preprocess.output.preprocessed_mnist.npz, train2.input.dataset]
- [preprocess.output.preprocessed_mnist.npz, train3.input.dataset]
Comments
0 comments
Please sign in to leave a comment.