In this section you will:
- Add a simple pipeline to your
valohai.yaml
. - Learn how to use different types of edges (optional).
Add a pipeline to valohai.yaml
The valohai.yaml configuration file is the blueprint for your project. To make sure the indentation is correct and all the steps contain the required properties, it is recommended to run the command
vh lint
in the directory where you have the valohai.yaml to check for any syntax errors.
Open the valohai.yaml
file and add the following pipeline
in it.
- pipeline:
name: preprocess-and-train
nodes:
- name: preprocess
type: execution
step: preprocess-dataset
- name: train
type: execution
step: train-model
edges:
- [preprocess.output.preprocessed_mnist.npz, train.input.preprocessed_dataset]
This pipeline will:
Create two nodes:
preprocess
will execute yourpreprocess-dataset
step.train
will execute yourtrain-model
step.
Create one edge to connect the nodes:
preprocessed_mnist.npz
frompreprocess-dataset
's outputs will be passed into the input calledpreprocessed_dataset
in thetrain
node.
You can now continue to the next part of the learning path. Optionally, you can keep reading to learn more about the different edge types.
Pipeline edges
The simplest syntax to define edges is [SOURCE
, TARGET
]
Both SOURCE
and TARGET
have 3 properties separated by dots: node, type and key. Thus the shorthand syntax becomes: [NODE
.TYPE
.KEY
, NODE
.TYPE
.KEY
]
- The available edge
types
are: -
- outputs (only source node)
- inputs (only target node)
- parameters
- metadata (only source node)
- files (only for deployment nodes)
In the example above you already saw how to pipe output from the source node into a target node input. Here are examples for using the other types of edges.
edges:
# Parameter-parameter edge
# Note that the parameters have to exist in the yaml for both source and target
- [sourcenode.parameter.parameterkey, targetnode.parameter.parameterkey]
# Metadata-parameter edge
# Note that you can only pass single values and not for example list type metadata
- [sourcenode.metadata.metadatakey, targetnode.parameter.parameterkey]
# Output-file edge
# The file type is only available for deployment nodes.
# Note that the filename is the name defined in the YAML.
- [sourcenode.output.outputkey, targetnode.file.endpointname.filename]
Next: Run a pipeline ⇒
Comments
0 comments
Please sign in to leave a comment.