How to choose the environment for different steps in a pipeline?

As with any Valohai executions, you can choose the environments you want to use for your pipeline steps. 

Option 1

You can change the environments for your pipeline steps in the Valohai UI when creating the pipeline. Click on the step in the pipeline graph and then choose the environment from the dropdown menu in the Runtime section.

Option 2

You can use the environment property in the step definition in you valohai.yaml. The value is the environment slug and you can get it by running the command vh environments on your CLI. 

When this property is not defined, the default environment set for your organization (or project) will be used. 

- step:
    name: preprocess-dataset
    image: python:3.9
    environment: aws-eu-west-1-g3s-xlarge
    command:
      - pip install numpy valohai-utils
      - python ./preprocess_dataset.py
    inputs:
      - name: dataset
        default: https://valohaidemo.blob.core.windows.net/mnist/mnist.npz
        

Option 3 

In some special cases you might want to override the environment for selected nodes in the pipeline definition in your valohai.yaml. 

- pipeline:
    name: Training Pipeline
    nodes:
      - name: preprocess
        type: execution
        step: preprocess-dataset
        override:
          environment: aws-eu-west-1-g3s-xlarge 
      - name: train
        type: execution
        step: train-model
      - name: evaluate
        type: execution
        step: batch-inference
    edges:
      - [preprocess.output.preprocessed_mnist.npz, train.input.dataset]
      - [train.output.model*, evaluate.input.model]