Configure an AWS S3 Bucket – Valohai Help

In this guide, we’ll link a private AWS S3 bucket to a Valohai project.

Requirements

an Amazon Web Services (AWS) account you can administer
a Valohai project which to link the S3 bucket to

Create the S3 bucket

Create an S3 bucket through AWS console (https://s3.console.aws.amazon.com/s3/home).

Select bucket name and region

Throughout this guide, we will assume the name of the bucket is my-valohai-bucket; be sure to replace this with the actual name of your bucket when copying in any example configuration!
Create the bucket in the region you’ll be running your training to minimize data transfer costs. If you don’t have a preference, we recommend using Ireland (eu-west-1) as most of our computation resides there.

Use default bucket properties & permissions

Default bucket properties are fine, but double-check that your bucket is not public. You can of course edit the default settings based on your needs.

Create a new bucket.

CORS Settings

If you wish to be able to upload files to the store using the app.valohai.com web UI, you will need to add a CORS policy document to the S3 bucket.

First, you navigate to the AWS S3 bucket you created.

Then you go to the Permissions tab and scroll down to Cross-origin resource sharing (CORS).

Click Edit add the rules below.

[
   {
      "AllowedHeaders": [
            "Authorization"
      ],
      "AllowedMethods": [
            "GET"
      ],
      "AllowedOrigins": [
            "*"
      ],
      "ExposeHeaders": [],
      "MaxAgeSeconds": 3000
   },
   {
      "AllowedHeaders": [
            "Authorization"
      ],
      "AllowedMethods": [
            "POST"
      ],
      "AllowedOrigins": [
            "https://app.valohai.com"
      ],
      "ExposeHeaders": [],
      "MaxAgeSeconds": 3000
   }
]

Now your bucket allows POSTs for your user on https://app.valohai.com website

Create an IAM user

Using the AWS console, start creating a new IAM user with programmatic access credentials (access key ID / secret access key).

Select name for your programmatic user

User name can be anything, try to be descriptive.
Double-check that programmatic access is turned on.

Skip the permission configuration

We will add permissions later, you can skip to the next step.

Save access key ID and secret for later usage

Download the CSV or copy-paste the “Access key ID” and “Secret access key” somewhere safe.

Tip

If you lose these credentials, you can generate new ones through IAM > Select user > Security credentials > Create access key.

Allow the IAM user to access the bucket

Now we have a user without any permissions, let’s allow the user to access our new bucket.

Find and open the user you created in the previous section.

Add a new inline policy. You can use any other AWS IAM policy definition methods just as well. Inline policies are the easiest get started.

The user needs to have full access to the S3 bucket; an example of a suitable access policy document is below. Make sure to change the resource name my-valohai-bucket!

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::my-valohai-bucket",
                "arn:aws:s3:::my-valohai-bucket/*"
            ]
        }
    ]
}

Give your policy a descriptive name and we are done with the mandatory AWS setup!

Large file uploads (optional)

If executions need to upload outputs larger than 5 GB, an additional setup is needed. This is optional and only required for large outputs.

To upload large outputs using Amazon’s multipart upload API, a temporary AWS IAM role will be dispensed to the worker machines when required.

Be sure to replace the following placeholders in the following policy examples!

my-valohai-bucket – the target S3 bucket
ACCOUNTNUMBER – your AWS account number
USERNAME – the username liked to the access keys that are being used with the store

You can find the username and account number by going to USERS and selecting the user you just created.

Click the copy button on the right of the User ARN.

Select the Roles tab and a create a new AWS IAM Role.

Select Custom trust policy.

The Custom trust policy document should look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNTNUMBER:user/USERNAME"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Replace the arn:aws:iam::ACCOUNTNUMBER:user/USERNAME with your own User ARN that you copied previously.

Click next.

Create policy for the role.

A new tab will open. Select JSON.

The policy JSON should look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "MultipartAccess",
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions",
                "s3:ListMultipartUploadParts",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-valohai-bucket",
                "arn:aws:s3:::my-valohai-bucket/*"
            ]
        }
    ]
}

Make sure to change the resource name my-valohai-bucket to your own bucket name.

Click Next: Tags and Next: Review.

Give your policy a descriptive name and click Create policy.

Go back to the tab that has the Role creating open and refresh the page.

Select the policy that you just created by clicking the checkbox in front of it.

Click next from the bottom of the page.

Give your role a descriptive name and click Create role from the bottom of the page.

Take note of the role’s AWS ARN (arn:aws:...), that will be configured to your Valohai project.

Link the store to Valohai

You can connect this data store either to a single project, or create in on the organization level. The recommended way is to create it under the organization. All the Data Stores added under the organization will be available for all projects.

Link to a Valohai organization

Login at https://app.valohai.com
Navigate to Hi, <name> (the top-right menu) > Manage <organization>.
Open the Data Stores tab.

The data store can be shared with everyone in the organization, or you can expose the data store only certain team(s).

Link the store to a Valohai organization:

Name your store and paste in the Bucket name and the IAM credentials in the fields provided.

If you also created the optional IAM Role for large uploads, paste the AWS ARN in the “Multipart Upload IAM Role ARN” field. Otherwise you may leave this field empty.

When you create the store, the credentials provided will be checked by creating a small test file in the bucket. The test file is automatically removed after the connection is verified.

Once you have created the store you can set the organization default store from the Data Stores view:

Link the store to a Valohai project

Navigate to Project > Settings > Data Stores > Add S3 store

Name your store and paste in the bucket name and the IAM credentials in the fields provided.

If you also created the optional IAM Role for large uploads, paste the ARN in the “Multipart Upload IAM Role ARN” field. You may leave this field empty.