- Tagging files allow you to easily find and sort data files
- You can use tag names when searching for an execution input file, or any data from your outputs
- In addition to tags, you can send arbitrary JSON metadata to be stored with your files.
Tag a file
You can add tags to the files either in the UI or during execution.
Web UI
Add tags in the Data tab of the project, or in executions outputs.
Programatically
You can save tags to your outputs by saving an additional JSON file .metadata.json
for each of your execution’s output files.
import valohai
import json
metadata = {
"valohai.tags": ["production", "lemonade"]
}
save_path = valohai.outputs().path('model.h5')
model.save(save_path)
metadata_path = valohai.outputs().path('model.h5.metadata.json')
with open(metadata_path, 'w') as outfile:
json.dump(metadata, outfile)
Find a file using tags
Once a file has been tagged you can easily find it in the UI.
- Create a new execution
- Add a new data file
- Search by tag name
You can also search for files by tags using the Valohai APIs (e.g. https://app.valohai.com/api/v0/data/?project=&tag=production
)
Learn more about the DatumList API.
Store arbitrary metadata with your files
You can attach additional metadata to each file you output from an execution by saving a .metadata.json
-file. The file should contain, in JSON format, all the metadata you’re looking to save with the file.
Valohai will store the contents of the .metadata.json-
file with the output file.
You can access the files metadata using Valohai DatumList API or DatumRetrieve API.
import valohai
import json
metadata = {
"valohai.tags": ["prod", "lemonade"], # creates Valohai tags for the file
"valohai.alias": "model-prod", # creates or updates a Valohai data alias to point to this output file
"key": "value",
"sample": "metadata"
}
save_path = valohai.outputs().path('model.h5')
model.save(save_path)
metadata_path = valohai.outputs().path('model.h5.metadata.json')
with open(metadata_path, 'w') as outfile:
json.dump(metadata, outfile)
Read a file's metadata
You can access the metadata that you’ve attached to a file either through the Valohai API or during execution.
Access files metadata during an execution
Any metadata created with datums is available during runtime under the /valohai/config/inputs.json
file.
import json
with open('/valohai/config/inputs.json') as json_file:
vh_inputs_config = json.load(json_file)
# Print metadata from each file that is in the input named "myinput"
for data in vh_inputs_config['myinput']['files']:
print(data["metadata"])
Comments
0 comments
Please sign in to leave a comment.