step.environment-variables

Environment variables are dynamic key-value pairs that running processes can write and read. This also holds true in the context of Valohai.

You can define environment variables in three ways:

1. Simply write export MY_KEY="my-value" in your commands. This is the most flexible way but won’t be tracked or easily searchable in Valohai.
2. Define expected environment variables in the valohai.yaml. You can define a default value that can be overwritten by the execution creator.
3. Define environment variables in the project settings. Any execution in the project can optionally be instructed to inherit the project environment variables. Project environment variables can be set to secret so they are not shown anywhere on the UI.

Environment variables in commands

You can define your environment variables in the commands like this:

- step:
    name: train-model
    image: python:3.6
    command:
    - export MODE="1"
    - export POST="clip"
    - python train.py

This is an acceptable approach if you are debugging, prototyping or the variables never change.

Environment variables in YAML

A better way is to explicitly define what environmental variables the execution expects:

- step:
    name: train-model
    image: python:3.6
    command: python train.py
    environment-variables:
      - name: MODE
        default: "1"
      - name: POST
        default: "clip"

A variable in environment-variables has four potential properties:

• name: The environment variable name how it will be passed to the execution.
• description: (optional) More detailed human-readable description of the variable.
• default: (optional) Default value for the environment variable.
• optional: (optional) Whether this environment variable is optional. All environment variables are optional by default so only optional: false would make sense.

This way the environment variables can easily be tracked and edited when creating executions either through the graphical user interface, command-line client or API.

Project level environment variables are excellent for private keys when doing more advanced data source integrations; or when you have some project-wide settings you want to share between executions.

Project environment variables

Make sure that the [X] Inherit project's environment variables and secrets checkbox is ticked when creating the execution.

Special environment variables

There are a handful of special environment variables that affect the behavior of the Valohai execution agent.

VH_NO_DATA_CACHE

Setting this to a true value (1, yes, true) will have the execution agent ignore any pre-existing cached data. This is useful if you have reused the same URL for a given datum (which, for reproducibility reasons, you should not generally do). The datum will still be written to the agent machine’s local cache.

VH_NO_IMAGE_CACHE

Setting this to a true value (1, yes, true) will have the execution agent ignore existing Docker images and force pulling them anew from the source.

VH_CLEAN

Setting this to a true value (1, yes, true) will have the execution agent forcibly clean all Docker images and cached data off the agent machine before and after the execution. This will take additional time.

VH_TMPFS

Setting this to a false value (0, no, false) will have the execution write all /tmp operations on the disk instead of the memory. A standard Valohai execution will say in the logs "/tmp is a memory filesystem" and write to memory when you write to /tmp. This might cause out of memory errors in some cases. Note: If you change the default behaviour and /tmp is written to disk, both reading and writing operations will be slower.

VH_SHM_SIZE

Will increase the size of the shared memory directory of the container. For example, if you'd like to increase the SHM size to 16GB then you can use 16G, 16 GB, 16 gb, or 16GB as the value