- Easily keep track of your execution's key metrics and sort executions based on your custom metrics.
- Anything that you print as JSON from your code gets collected as potential Valohai metadata.
- Visualize and compare execution metrics as a time series, a scatter plot or a confusion matrix in the UI.
- You can set early stopping rules for executions and Tasks or stop a pipeline from proceeding based on metadata values.
- You can download the metadata as a CSV or JSON file from the executions metadata tab.
To create visualizations, you can either:
-
save them as images to
/valohai/outputs
to be uploaded -
use our metadata system to render interactive graphs on the web interface
Visualizations from Metadata
Execution metadata is output by writing lines of JSON to the standard output stream.
Collect metadata with valohai-utils
import valohai
with valohai.metadata.logger() as logger:
logger.log("step", epoch)
logger.log("accuracy", accuracy)
logger.log("loss", loss)
Collect metadata with Python without valohai-utils
import json
print()
print(json.dumps({
"step": epoch,
"accuracy": accuracy,
"loss": loss
}))
Collect metadata with R
library(jsonlite)
metadataEvent <- list()
metadataEvent[["step"]] <- epoch
metadataEvent[["accuracy"]] <- accuracy
metadataEvent[["loss"]] <- loss
write(toJSON(metadataEvent, auto_unbox=TRUE), stdout())
Each metadata point also has an implicit value _time
that tells when the metadata line was output. The _time
value is in UTC, formatted as an ISO-8601 DateTime (e.g. 2017-04-04T15:03:39.321000
).
You can generate real-time charts based on metadata which helps with monitoring long runs so you can stop them if training doesn’t converge well. Optionally, you can use any metadata value as the Horizontal axis value by selecting it in the corresponding dropdown menu.
You can add and remove graphs for different metadata by clicking on “Add/remove”. Once added, you can smooth the graph and choose the position of the vertical axis for each metadata.
You can sort executions by metadata values in the web interface under the Executions tab which is useful for e.g. finding training executions with the highest prediction accuracy.
The latest or last value of each key such as accuracy
can be used for the sorting hyperparameter optimization results.
What else can I do with metadata?
Beyond creating the visuals, based on metadata values you can for example:
-
Easily sort execution tables.
-
Easily compare executions.
-
Set early-stopping rules for executions and Tasks.
-
Set pipeline conditions.
Comments
0 comments
Please sign in to leave a comment.