Pipeline parameters

In some cases you might have parameters that are shared between several nodes in the pipeline. Instead of having to go and update each node separately when the parameter values are changed, you can use pipeline parameters. They will override the default values defined in the valohai.yaml, including cases where the default value is left empty. 

Defining pipeline parameters in valohai.yaml

Similarly to step parameters, pipeline parameters are added in the valohai.yaml file. You will need to specify the targets for them, i.e. the step parameters that they will override. 

- step:
    name: pre_step
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./preprocess.py
    parameters:
      - name: exec_id
        type: string
      - name: pre_param
        type: integer
        default: 5
    inputs:     
      - name: pre_dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz
        
- step:
    name: train_step
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./train.py
    parameters:
      - name: exec_id
        type: string
      - name: train_param
        type: integer
        default: 5
         inputs:
    inputs:     
      - name: dataset
        
- pipeline:
    name: Example for Pipeline Parameters
    parameters:
      - name: id
        targets:
          - preprocess.parameters.exec_id
          - train.parameters.exec_id
      - name: training_parameter
        targets:
          - train.parameters.train_param
        default: 3
      - name: task_parameter
        target: train_in_task.parameters.train_param
        default: 123
    nodes:
      - name: preprocess
        step: pre_step
        type: execution
      - name: train
        step: train_step
        type: execution
      - name: train_in_task
        step: train_step
        type: task
    edges:
      - [preprocess.output.preprocessed_mnist.npz, train.input.dataset]
      - [preprocess.output.preprocessed_mnist.npz, train_in_task.input.dataset]

In the example above

• The syntax for defining pipeline parameter targets is <node-name>.parameters.<step-parameter-name>.
• The pipeline parameter id gets two targets: preprocess.parameters.exec_id and train.parameters.exec_id. This means that the pipeline parameter value overrides the exec_id value in both preprocess and train nodes.
• The exec_id for the train_in_task node is not affected by the pipeline parameter id but gets the default value from the step definition. 
• Note that it is not possible to access the value of the pipeline parameter but you will need to read the actual value from the node, e.g valohai.parameters("exec_id").value. 

Using pipeline parameters

In the UI

The picture below shows the parameter options for the preprocess node from the previous valohai.yaml example.

• You can see the target parameters and default values for each pipeline parameter under the Shared Parameters section in the UI. You can also define new values for the pipeline parameters in this section.
• The parameter exec_id will be overridden by the pipeline parameter id and its value cannot be changed under the node's parameter section.
  • Note that the pipeline parameter id is empty in the picture below as there was no default value for it in the valohai.yaml.
• The parameter pre_param has not been defined as a target for any pipeline parameter so its value can be edited under the node parameters. 
•  Note that here the train_in_task node has been defines as a Task node which allows defining multiple values for the pipeline parameter. 

With an API call

The easiest way to get a template for your API call payload is to first create a pipeline in the UI and then click on the Show as API call button at the bottom of the page next to the Create pipeline button. 

The example below shows how the pipeline parameters would look for the example pipeline presented here. Note that this is not the full payload and you will need to also define for example the nodes and edges in it. 

• Pay attention to how the expression is defined for single and multiple type parameters ("style": "single" and "style": "multiple", respetively). 

"parameters": {
            "id": {
                "config": {
                    "name": "id",
                    "targets": [
                        "preprocess.parameters.exec_id",
                        "train.parameters.exec_id"
                    ]
                },
                "expression": {
                    "style": "single",
                    "rules": {
                        "value": ""
                    }
                }
            },
            "training_parameter": {
                "config": {
                    "name": "training_parameter",
                    "default": 3,
                    "targets": [
                        "train.parameters.train_param"
                    ]
                },
                "expression": {
                    "style": "single",
                    "rules": {
                        "value": 3
                    }
                }
            },
            "task_parameter": {
                "config": {
                    "name": "task_parameter",
                    "default": 123,
                    "targets": [
                        "train_in_task.parameters.train_param"
                    ]
                },
                "expression": {
                    "style": "multiple",
                    "rules": {
                        "items": [
                            123
                        ]
                    }
                }
            }
        }