Can I have multiple valohai.yaml configuration files?

Yes. Valohai supports monorepositories.

Each Valohai project can be connected to one valohai.yaml configuration file. That file can be either in the root of the repository or in a subfolder.

By default, Valohai will look for the valohai.yaml configuration file in the root of the repository. You can define a different valohai.yaml file by going to your Project -> Settings -> Repository, and define the path your valohai.yaml file.

For example, if your configuration file is inside a subfolder called "ingestion", then you'd define the path as ingestion/valohai.yaml.

If you're running an adhoc job, before commiting the changes and changing the yaml path in the UI you can set the yaml path with the --yaml flag. For example, for the case above where the valohai.yaml is in the "ingestion" subfolder, you would run:

vh exec run mystep --adhoc --yaml ingestion/valohai.yaml

When you're ready to commit the changes make sure to change the path in the Valohai UI as well. 

Running non-adhoc execution from the CLI 

In case you're running non-adhoc execution from the CLI in a project with a custom path for your valohai.yaml, you will need to either set the project-mode to remote or define the commit you want to use. 

vh --project-mode remote --project <project-id> exec run <step-name>

or

vh exec run <step-name> --commit <commit-id>

When using the project-mode option, the latest fetched commit fill be used. You can of course also use the --commit flag with this command to use a specific commit instead.