Mounting gives you direct access to the network file system like GCP Filestore, without having to download the files on your machine.
Valohai does not version or keep track of the changes made inside the mounted file system. The files that are created, edited, or deleted will not be versioned as part of a Valohai execution.
We strongly recommend using the Valohai inputs and outputs system, as they are versioned as a part of the execution.
You can mount a filesystem to access a large dataset, run preprocessing operations, and output the processed dataset into Valohai Outputs before it’s used further in the pipeline. This way the snapshot of the preprocessed data will be versioned.
Define a mount in YAML
You can either use an existing or create a new GCP Filestore.
- Create your Filestore in the same VPC where all Valohai resources are in or set up VPC peering between the two VPCs
- step: name: mount-sample image: python:3.7 command: - ls -la /my-data mounts: - destination: /my-data source: <IP>:/mystore type: nfs readonly: false
Please sign in to leave a comment.