Attach tags and metadata to output files – Valohai Help

Tagging files allow you to easily find and sort data files
You can use tag names when searching for an execution input file, or any data from your outputs
In addition to tags, you can send arbitrary JSON metadata to be stored with your files.

Tag a file

You can add tags to the files either in the UI or during execution.

Web UI

Add tags in the Data tab of the project, or in executions outputs.

Programatically

You can save tags to your outputs by saving an additional JSON file .metadata.json for each of your execution’s output files.

import valohai
import json

metadata = {
    "valohai.tags": ["production", "lemonade"]
}

save_path = valohai.outputs().path('model.h5')
model.save(save_path)

metadata_path = valohai.outputs().path('model.h5.metadata.json')
with open(metadata_path, 'w') as outfile:
    json.dump(metadata, outfile)

Find a file using tags

Once a file has been tagged you can easily find it in the UI.

Create a new execution
Add a new data file
Search by tag name

You can also search for files by tags using the Valohai APIs (e.g. https://app.valohai.com/api/v0/data/?project=&tag=production)

Learn more about the DatumList API.

Store arbitrary metadata with your files

You can attach additional metadata to each file you output from an execution by saving a .metadata.json-file. The file should contain, in JSON format, all the metadata you’re looking to save with the file.

Valohai will store the contents of the .metadata.json-file with the output file.

You can access the files metadata using Valohai DatumList API or DatumRetrieve API.

import valohai
import json

metadata = {
    "valohai.tags": ["prod", "lemonade"], # creates Valohai tags for the file
    "valohai.alias": "model-prod", # creates or updates a Valohai data alias to point to this output file
    "key": "value",
    "sample": "metadata"
}

save_path = valohai.outputs().path('model.h5')
model.save(save_path)

metadata_path = valohai.outputs().path('model.h5.metadata.json')
with open(metadata_path, 'w') as outfile:
    json.dump(metadata, outfile)

Read a file's metadata

You can access the metadata that you’ve attached to a file either through the Valohai API or during execution.

Access files metadata during an execution

Any metadata created with datums is available during runtime under the /valohai/config/inputs.json file.

import json

with open('/valohai/config/inputs.json') as json_file:
    vh_inputs_config = json.load(json_file)

# Print metadata from each file that is in the input named "myinput"
for data in vh_inputs_config['myinput']['files']:
    print(data["metadata"])