Pipeline Tasks is an early access feature. If you're interested in trying it out, let your Valohai contact know or email us at email@example.com.
It is possible to run a pipeline inside a Task. One of the advantages is that this allows you to connect two consecutive Task executions instead of running two complete Tasks which would result in all the outputs from the first Task to be passed to all executions on the second one.
Example use case could be for example:
- Running pipelines over a range of customer or product ids.
The main pre-requisite for running a pipeline in a Task is that you have defined at least one pipeline parameter in your
valohai.yaml. The pipeline parameters will override the default values, including cases where the default value is left empty.
When it comes to pipeline Tasks, it is currently possible to give only single values to the pipeline nodes via pipeline parameters. If you want to have a Task node inside the pipeline, you will have to do that by choosing the node and inputing the parameters as you would for a single pipeline (see the gif below).
Running a pipeline Task in the UI
Below, you can find a simple example for a pipeline with pipeline parameters.
- step: name: pre_step image: python:3.10 command: - pip install valohai-utils - python ./preprocess.py parameters: - name: exec_id type: string default: "id_123" - name: pre_param type: integer default: 5 inputs: - name: pre_dataset default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz - step: name: train_step image: python:3.10 command: - pip install valohai-utils - python ./train.py parameters: - name: exec_id type: string default: "id_123" - name: train_param type: integer default: 5 inputs: - name: dataset default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz optional: true - pipeline: name: Example for Pipeline Task parameters: - name: id targets: - preprocess.parameters.exec_id - train.parameters.exec_id default: "id_123" nodes: - name: preprocess step: pre_step type: execution - name: train step: train_step type: execution override: inputs: - name: dataset edges: - [preprocess.output.preprocessed_mnist.npz, train.input.dataset]
Remember to commit and push your changes to Git and fetch them to your Valohai project!
- Open the Task tab of your project.
- Click on the Create Task button.
- Choose Pipeline as the Task Child Type from the dropdown menu at the top of the page.
- Choose the blueprint for the pipeline you want to run.
- Click on the Create Pipeline from Template button.
- Choose the Task type and other Task properties.
Under Variant parameters, Set the values for each parameter
- Choose the type: Single, Multiple, Linear, Logspace or Random
- Input the values (for multiple input one value per line)
- Click on the Create Task button
Please sign in to leave a comment.