How can I use a whole folder as an input for an execution?

Valohai inputs can be a single file or multiple files. You can either list the files you want for your input or use a wildcard to download all the files.

- step:
    name: train
    image: tensorflow/tensorflow:2.6.0
    command: python train.py
    inputs:
    - name: model
      default:
        - s3://mybucket/models/model.pb
    - name: images
      default:
        - s3://mybucket/images/*.jpg
      keep-directories: suffix

keep-directories is used to define what folder structure should Valohai use in the inputs folder.

• none: (default) all files are downloaded to /valohai/inputs/myinput
• full: keeps the full path from the storage root. For example s3://special-bucket/foo/bar/**.jpg could end up as /valohai/inputs/myinput/foo/bar/dataset1/a.jpg
• suffix: keeps the suffix from the “wildcard root”. For example s3://special-bucket/foo/bar/* the special-bucket/foo/bar/ would be removed, but any relative path after it would be kept, and you might end up with /valohai/inputs/myinput/dataset1/a.jpg