Introduction to metadata, metrics, and visualizations

To create visualizations, you can either:

1. save them as images to /valohai/outputs to be uploaded

2. use our metadata system to render interactive graphs on the web interface

Visualizations from Metadata

Execution metadata is output by writing lines of JSON to the standard output stream.

Collect metadata with valohai-utils

import valohai

with valohai.metadata.logger() as logger:
    logger.log("step", epoch)
    logger.log("accuracy", accuracy)
    logger.log("loss", loss)

Collect metadata with Python without valohai-utils

import json

print()
print(json.dumps({
    "step": epoch,
    "accuracy": accuracy,
    "loss": loss
}))

Collect metadata with R

library(jsonlite)

metadataEvent <- list()
metadataEvent[["step"]] <- epoch
metadataEvent[["accuracy"]] <- accuracy
metadataEvent[["loss"]] <- loss

write(toJSON(metadataEvent, auto_unbox=TRUE), stdout())

Each metadata point also has an implicit value _time that tells when the metadata line was output. The _time value is in UTC, formatted as an ISO-8601 DateTime (e.g. 2017-04-04T15:03:39.321000).

You can generate real-time charts based on metadata which helps with monitoring long runs so you can stop them if training doesn’t converge well. Optionally, you can use any metadata value as the Horizontal axis value by selecting it in the corresponding dropdown menu.

You can add and remove graphs for different metadata by clicking on “Add/remove”. Once added, you can smooth the graph and choose the position of the vertical axis for each metadata.

You can sort executions by metadata values in the web interface under the Executions tab which is useful for e.g. finding training executions with the highest prediction accuracy.

The latest or last value of each key such as accuracy can be used for the sorting hyperparameter optimization results.

What else can I do with metadata?

Beyond creating the visuals, based on metadata values you can for example:

• Easily sort execution tables.

• Easily compare executions.

• Set early-stopping rules for executions and Tasks.

• Set pipeline conditions.