Edges connect the source and target nodes to each other in pipelines. They are often defined by setting the output of the source node to be the input of the target node.
In addition, it is possible to define the pipeline edges with parameters. This means that the chosen parameter from the source node will be passed to a parameter in the target node. The passed value will be used during the execution of the target node instead of the default value from valohai.yaml.
Similarly, it is possible to collect metadata in the source node and pass it into a parameter in the target node.
Note that in both cases, the parameter where the values are passed has to be defined in the yaml for the target node.
Example:
- step:
name: source-node
image: tensorflow/tensorflow:2.5.0
command:
- pip install valohai-utils
- python mycode.py {parameters}
parameters:
- name: param1
default: one
description: This will be passed to target-node
type: string
- step:
name: target-node
image: tensorflow/tensorflow:2.5.0
command:
- pip install valohai-utils
- python myothercode.py {parameters}
parameters:
- name: param1
default: one
description: This will retain its value
type: string
- name: param2
default: two
description: In the pipeline, the value will be replace by param1 value from source-node
type: string
- name: param3
default: three
description: In the pipeline, the value will be replace by mymetadata value from source-node
type: string
- pipeline:
name: Parameter edges
edges:
- configuration: {}
source: source-node.parameter.param1
target: target-node.parameter.param2
- configuration: {}
source: source-node.metadata.mymetadata
target: target-node.parameter.param3
nodes:
- name: source-node
override: {}
step: source-node
type: execution
- name: target-node
override: {}
step: target-node
type: execution
Comments
0 comments
Please sign in to leave a comment.