Introduction to the valohai.yaml configuration file

Each Valohai-enabled git repository has one or multiple valohai.yaml configuration files that define the runs that can be executed and what kind of REST endpoints the project can automatically generate. In the next section, we’ll cover these execution types or “steps”, as we call them.

A single configuration file can be used by multiple projects by various users, as long as the user has access to the git repository.

Example of a valohai.yaml file:

# This step downloads a set of images
# and runs preprocess.py
- step:
    name: Preprocess dataset (MNIST)
    image: tensorflow/tensorflow:1.13.1-py3
    command: python preprocess.py
    inputs:
      - name: training-set-images
        default: https://valohaidemo.blob.core.windows.net/mnist/imgages.tar.gz

# This step downloads preprocessed-data
# and runs train.py with two parameters (max_steps and learning_rate)
- step:
    name: Train model
    image: tensorflow/tensorflow:1.13.1-py3
    command: python train.py {parameters}
    parameters:
      - name: max_steps
        type: integer
        default: 300
      - name: learning_rate
        type: float
        default: 0.001
    inputs:
      - name: preprocessed-data
        default: https://valohaidemo.blob.core.windows.net/mnist/data.tar.gz

# This step downloads a trained model file and images,
# installs libraries listed in requirements.txt
# and runs batch_inference.py
- step:
    name: Batch inference
    image: tensorflow/tensorflow:1.13.1-py3
    command:
      - pip install -r requirements.txt
      - python batch_inference.py
    inputs:
      - name: model
      - name: images

Large YAML files

Each Project can have one valohai.yaml file. As your project grows, you might start having repetitive content in your YAML file.

We recommend using YAML anchors, aliases, and extensions to define and re-use repeatitive content.

- definitions:

    my-inputs: &my_inputs

        - name: test-set
          keep-directories: suffix
          default: s3://onboard-sample/test/*
        - name: train-set
          keep-directories: suffix
          default: s3://onboard-sample/train/*

- step:
    name: train-model
    image: tensorflow/tensorflow:2.0.1
    command:
      - pip install valohai-utils numpy
      - python ./train.py {parameters}
    parameters:
      - name: epoch
        default: 5
        type: integer

    inputs: *my_inputs