Datum adoption

In general, only files that go through execution or are uploaded into Valohai will get a datum URL. However, in some cases, it might be beneficial to create datum URLs for existing files in your S3 storage without needing to push them through Valohai separately.

To do this, it is possible to adopt storage URLs into real, Valohai-tracked datums. This gives all the benefits related to datum URLs, including traceability and setting aliases.

It is possible to adopt storage URLs into Valohai datums either in the web UI or by using the Valohai APIs.

In the Web UI

• Open your project on app.valohai.com
• Open the Data tab
• Open the Adopt tab
• Select the Destination store from the dropdown menu
• Add the URLs to adopt (one per line). Note that they have to be supported by the destination store.
• Click Adopt selected files

By using API requests

• Generate yourself an API token as shown in API Basics
• Send a POST request to https://app.valohai.com/api/v0/stores/{id}/adopt/
  • You can get the IDs to replace the placeholder {id} for all your stores by sending an API request to https://app.valohai.com/api/v0/stores/
• In the request body, send a JSON consisting of the URL to adopt and the project ID:
  • You can get the project ID from under the Settings tab in the Valohai UI

{
    "url":"s3://project-name/samplefile.txt",
    "project":"project ID"
}

If the datum adoption was successful, you should get the following response.

{
    "ok": true,
    "created": {
        "s3://my-bucket/my-file.txt": "017a515f-30a4-d0f1-d37a-53ffc38e90c7"
        }
}

In case the datum already exists, this will be indicated in the response.

{
    {
        "message": "s3://my-bucket/my-file.txt already exists in my-bucket",
        "code": null
    }
}

Finally, if the file was not found, you will be notified of it as well.

{
    {
        "message": "Not found in my-bucket: 's3://my-bucket/my-file.txt'",
        "code": "adoptable_file_not_found"
    }
}