Save and upload files from executions

• You can output any type of files from Valohai. For example trained models, CSV-files, images, or something else.
• Valohai outputs will be saved to the project's default data store (AWS S3, Azure Storage, GCP Cloud Storage, OpenStack Swift).
• By default, all outputs will be saved to your data store at the end of an execution. You can use Live outputs to save files mid-execution.
• Each output will receive a unique datum:// link that you can use to download the file to another execution. If you’re using your own data store, you’ll also receive a link specific to that data store.

Write your files to the /valohai/outputs/ folder to save and upload them to your cloud storage. The saved files will appear under the Outputs tab of your execution and in the projects Data tab.

Save output files with valohai-utils

import valohai

# Define inputs available for this step and their default location
# The default location can be overriden when you create a new execution (UI, API or CLI)
default_inputs = {
    'myinput': 's3://bucket/mydata.csv'
}

# Create a step 'train' in valohai.yaml with a set of inputs
valohai.prepare(step="train", image="tensorflow/tensorflow:2.6.1-gpu", default_inputs=default_inputs)

# Open the CSV file from Valohai inputs
with open(valohai.inputs("myinput").path()) as csv_file:
    reader = csv.reader(csv_file, delimiter=',')

Save output files in Python without valohai-utils

# Get the location of Valohai inputs directory
VH_INPUTS_DIR = os.getenv('VH_INPUTS_DIR', '.inputs')

# Get the path to your individual inputs file
# e.g. /valohai/inputs/<input-name>/<filename.ext>
path_to_file = os.path.join(VH_INPUTS_DIR, 'myinput/mydata.csv')

pd.read_csv(path_to_file)

Save output files in R

# Get the location of Valohai inputs directory
vh_inputs_dir <- Sys.getenv("VH_INPUTS_DIR", unset = ".inputs")

# Get the path to your individual inputs file
# e.g. /valohai/inputs/<input-name>/<filename.ext>
path_to_file <- file.path(vh_inputs_dir, "myinput/mydata.csv")

import_df <- read.csv(path_to_file, stringsAsFactors = F)

You can then use the copied datum:// link as an input in another execution. For example, use a link to a trained model file in a new Batch Inference execution.