Run a pipeline Task

It is possible to run a pipeline inside a Task. One of the advantages is that this allows you to connect two consecutive Task executions instead of running two complete Tasks which would result in all the outputs from the first Task to be passed to all executions on the second one. 

Example use case could be for example:

• Running pipelines over a range of customer or product ids. 

The main pre-requisite for running a pipeline in a Task is that you have defined at least one pipeline parameter in your valohai.yaml. The pipeline parameters will override the default values, including cases where the default value is left empty. 

Running a pipeline Task in the UI

Below, you can find a simple example for a pipeline with pipeline parameters. 

- step:
    name: pre_step
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./preprocess.py
    parameters:
      - name: exec_id
        type: string
        default: "id_123"
      - name: pre_param
        type: integer
        default: 5
    inputs:
      - name: pre_dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz
        
- step:
    name: train_step
    image: python:3.10
    command:
      - pip install valohai-utils
      - python ./train.py
    parameters:
      - name: exec_id
        type: string
        default: "id_123"
      - name: train_param
        type: integer
        default: 5
    inputs:
      - name: dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz
        optional: true

- pipeline:
    name: Example for Pipeline Task
    parameters:
      - name: id
        targets:
          - preprocess.parameters.exec_id
          - train.parameters.exec_id
        default: "id_123"
    nodes:
      - name: preprocess
        step: pre_step
        type: execution
      - name: train
        step: train_step
        type: execution
        override:
          inputs:
            - name: dataset
    edges:
      - [preprocess.output.preprocessed_mnist.npz, train.input.dataset]

Remember to commit and push your changes to Git and fetch them to your Valohai project! 

• Open the Task tab of your project.
• Click on the Create Task button.
• Choose Pipeline as the Task Child Type from the dropdown menu at the top of the page.
• Choose the blueprint for the pipeline you want to run.
• Click on the Create Pipeline from Template button.

• Choose the Task type and other Task properties. 
• Under Variant parameters, Set the values for each parameter
  • Choose the type: Single, Multiple, Linear, Logspace or Random
  • Input the values (for multiple input one value per line)
• Click on the Create Task button