In some cases you might have parameters that are shared between several nodes in the pipeline. Instead of having to go and update each node separately when the parameter values are changed, you can use pipeline parameters. They will override the default values defined in the valohai.yaml
, including cases where the default value is left empty.
Defining pipeline parameters in valohai.yaml
Similarly to step parameters, pipeline parameters are added in the valohai.yaml file. You will need to specify the targets for them, i.e. the step parameters that they will override.
- step:
name: pre_step
image: python:3.10
command:
- pip install valohai-utils
- python ./preprocess.py
parameters:
- name: exec_id
type: string
- name: pre_param
type: integer
default: 5
inputs:
- name: pre_dataset
default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz
- step:
name: train_step
image: python:3.10
command:
- pip install valohai-utils
- python ./train.py
parameters:
- name: exec_id
type: string
- name: train_param
type: integer
default: 5
inputs:
inputs:
- name: dataset
- pipeline:
name: Example for Pipeline Parameters
parameters:
- name: id
targets:
- preprocess.parameters.exec_id
- train.parameters.exec_id
- name: training_parameter
targets:
- train.parameters.train_param
default: 3
- name: task_parameter
target: train_in_task.parameters.train_param
default: 123
nodes:
- name: preprocess
step: pre_step
type: execution
- name: train
step: train_step
type: execution
- name: train_in_task
step: train_step
type: task
edges:
- [preprocess.output.preprocessed_mnist.npz, train.input.dataset]
- [preprocess.output.preprocessed_mnist.npz, train_in_task.input.dataset]
In the example above
- The syntax for defining pipeline parameter targets is
<node-name>.parameters.<step-parameter-name>
. - The pipeline parameter
id
gets two targets:preprocess.parameters.exec_id
andtrain.parameters.exec_id
. This means that the pipeline parameter value overrides theexec_id
value in bothpreprocess
andtrain
nodes. - The
exec_id
for thetrain_in_task
node is not affected by the pipeline parameterid
but gets the default value from the step definition. - Note that it is not possible to access the value of the pipeline parameter but you will need to read the actual value from the node, e.g
valohai.parameters("exec_id").value
.
Using pipeline parameters
In the UI
The picture below shows the parameter options for the preprocess node from the previous valohai.yaml example.
- You can see the target parameters and default values for each pipeline parameter under the Shared Parameters section in the UI. You can also define new values for the pipeline parameters in this section.
- The parameter
exec_id
will be overridden by the pipeline parameterid
and its value cannot be changed under the node's parameter section.- Note that the pipeline parameter
id
is empty in the picture below as there was no default value for it in thevalohai.yaml
.
- Note that the pipeline parameter
- The parameter
pre_param
has not been defined as a target for any pipeline parameter so its value can be edited under the node parameters. - Note that here the
train_in_task
node has been defines as a Task node which allows defining multiple values for the pipeline parameter.
With an API call
The easiest way to get a template for your API call payload is to first create a pipeline in the UI and then click on the Show as API call button at the bottom of the page next to the Create pipeline button.
The example below shows how the pipeline parameters would look for the example pipeline presented here. Note that this is not the full payload and you will need to also define for example the nodes and edges in it.
- Pay attention to how the expression is defined for single and multiple type parameters (
"style": "single"
and"style": "multiple"
, respetively).
"parameters": {
"id": {
"config": {
"name": "id",
"targets": [
"preprocess.parameters.exec_id",
"train.parameters.exec_id"
]
},
"expression": {
"style": "single",
"rules": {
"value": ""
}
}
},
"training_parameter": {
"config": {
"name": "training_parameter",
"default": 3,
"targets": [
"train.parameters.train_param"
]
},
"expression": {
"style": "single",
"rules": {
"value": 3
}
}
},
"task_parameter": {
"config": {
"name": "task_parameter",
"default": 123,
"targets": [
"train_in_task.parameters.train_param"
]
},
"expression": {
"style": "multiple",
"rules": {
"items": [
123
]
}
}
}
}
Comments
0 comments
Please sign in to leave a comment.