Configure OpenStack Swift

In this guide, we’ll link a private OpenStack Swift container to a Valohai project.

Requirements

• an OpenStack account you can administer
• a Valohai project which to link the Swift container to
• any modern version of Python

Install OpenStack clients

First we need to install OpenStack and Swift command-line clients.

pip install python-openstackclient python-swiftclient

Then we need to authenticate to your OpenStack installation. The easiest way to authenticate is using an OpenStack RC File if your provider allows generating those. More authentication methods in OpenStack documentation.

For example:

Run the OpenStack RC File:

source project-1234-openrc.sh # it will ask your password but you can optionally edit the .sh file # to include your password automatically by editing the OS_PASSWORD variable

Now your OpenStack client is ready for use.

Create a container

Create a Swift container for your Valohai project.

openstack container create my-project-data
+----------------+-----------------+------------+
| account        | container       | x-trans-id |
+----------------+-----------------+------------+
| AUTH_123456789 | my-project-data | 1234567890 |
+----------------+-----------------+------------+

openstack container show my-project-data
+--------------+--------------------------------+
| Field        | Value                          |
+--------------+--------------------------------+
| account      | AUTH_123456789                 |
| bytes_used   | 0                              |
| container    | my-project-data                |
| object_count | 0                              |
+--------------+--------------------------------+

Configure Temporary URL key

Temporary URL keys are required to give secure download privileges for Valohai workers.

# generate random string from somewhere to use as Temporary URL key, e.g.
cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1
# set the key to the Swift cluster
swift post -m "Temp-URL-Key:randomlygeneratedkey"

Configure CORS

If you wish to be able to upload files to the store using the Valohai web user interface, you will need to add a CORS (Cross-Origin Resource Sharing) policy to the Swift container.

openstack container set \
    --property Access-Control-Allow-Origin='https://app.valohai.com' \
    --property Access-Control-Max-Age='3000' \
    my-project-data

swift stat -v my-project-data
# ...
Meta Access-Control-Allow-Origin: https://app.valohai.com
Meta Access-Control-Max-Age: 3000
# ...

Now your container allows uploads from https://app.valohai.com website

Link the store to Valohai

You can connect this data store either to a single project, or create in on the organization level.

Link to a Valohai organization

1. Login at https://app.valohai.com
2. Navigate to Hi, <name> (the top-right menu) > Manage <organization>.
3. Open the Data Stores tab and add your store's details

The data store can be shared with everyone in the organization, or you can expose the data store only certain team(s).

Link the store to a Valohai project

Navigate to Project > Settings > Data Stores > Add OpenStack Swift Object Store

Required fields:

• Name: Understandable name for the storage.
• Auth URL: This can be found in the OpenStack RC File, variable OS_AUTH_URL.
• Auth Version: Valohai currently only supports 3.0. Contact support for more information.
• Username: Your OpenStack account username.
• Password: Your OpenStack account password.
• Container Name: The name of the container. We used my-project-data in this guide.
• Region Name: openstack region list gives you a list of all the regions for your installation.
• emp URL Key: This is the key we previously generated in step 3.1.

Once the store has been set up, you can set the store as your project’s default store in the Settings > General view. This ensures outputs will be stored in your Swift container.