You use live outputs to upload files mid-execution.
Normally, executions will upload all files stored at /valohai/outputs
at the end of the execution, even if the code crashes or is stopped. But in some long running workloads, you might want to save checkpoints or other artifacts mid-execution.
When Valohai detects that a file under /valohai/outputs
is marked as read-only, Valohai will remove that file from the directory and upload it right away. All programming languages and shells have a way for marking files read-only e.g. os.chmod
in Python.
A simple example:
echo hello >> /valohai/outputs/greeting.txt
chmod 0444 /valohai/outputs/greeting.txt
sleep 30
echo bye >> /valohai/outputs/farewell.txt
# => Generates 2 files:
# - 'greeting.txt' with 'hello', uploaded right away
# - 'farewell.txt' with 'bye', uploaded after the execution finishes
If the file permissions change commands are quick, they might not register before the code continues its progress. Because of this, we advice not to generate multiple live outputs with exactly the same name.
That can lead to some funky behaviour.
Comments
0 comments
Please sign in to leave a comment.