In addition to using the Ecosystem steps provided by Valohai, you can also create and add your own library steps.

This example shows you how to create a custom library steps that take in a Dockerfile, use it to build a docker image and finally push the image to a private registry. The files used in the example are available in the public valohai-toolkit Github repository. There are three different steps for different image registries (AWS ECR, GCP Artifact Registry and Docker Hub). 

• The idea behind the example is that writing a Dockerfile is relatively straightforward for a lot of cases but setting up Docker, finding your organisation registry credentials, and learning how to authenticate with the whole process is more time consuming. 

Prerequisites

• A Git repository containing a valohai.yaml with at least one step and all the other required code files.
• A Docker registry.
  • This example covers three use cases for private registries (Docker Hub login with credentials, AWS ECR login with machine identity, GCP Artifact Registry login with service account key). 
• Admin account in your Valohai organization. 
• A Valohai environment with the docker run argument --privileged. Contact Valohai Support (support@valohai.com) if you need one added to your organization.
  • Note that privileged containers have root capabilities on the host. This is required for running the Docker daemon inside a Docker container in this example. 

Connect to a public repository

The Docker Image Builder steps are available in the public valohai-toolkit repository. You can use these instructions to connect also your own repositories. 

• Login to app.valohai.com.
• Click on Hi, <username>! on the top right corner.
• Select Manage <organization>.
• Open the Libraries tab.
• Click on the Add a new Library button.
• Name your library.
  • Note that the name can only consist of  letters, numbers, underscores or hyphens.
• Add HTTPS URL of the public repository in the respective field.
• Click on the Add Library button.
• Click on the Update button to fetch the commits.

The steps defined in the valohai.yaml file are now available under the Create Execution page of all your projects!

Build and push an image to a Docker registry

Once you have connected the repository as a Library, you can follow the steps below to build and push an image to your Docker registry. 

• Generate an execution by utilizing the library. 
  • Click on the Create Execution button on the Executions tab of your project and choose the step you want to run from the custom library. 
• Configure the Dockerfile (input or parameter) and adjust other parameters.
• Include the needed environmental variables for authentication as explained below.

Configure the Dockerfile

You can either provide the Dockerfile as in input from your data store or write it in the dockerfile parameter in the Valohai UI. By default, if the input file exists it will be user instead of the parameter. 

The default values in this example will result in a Docker image with Python 3.9, valohai-utils and tensorflow 2.6.0 installed.

Define the parameter values

You will need to provide the following parameters:

• docker-tag - defines the image tag when building and pushing the image.
  • E.g. 0.1 would create an image username/myimagename:0.1. Note that the first part comes from the repository parameter. 
• repository - Name of your repository
  • E.g. username/myimagename

Define the environment variables

Depending on which registry you are using, you will need to provide certain environment variables. You can define these either on the execution, project or organization level (admin users only) depending on the use case.  

• Docker Hub
  • DHUSERNAME- Your Docker Hub username
  • DHPASSWORD- Your Docker Hub password. Make sure to save it as a secret! 
• AWS ECR
  • AWS_ECR_ACCOUNT_ID - ID of your AWS account that hosts of the Elastic Container Registry.
  • AWS_ECR_REGION - AWS Region for the Elastic Container Registry, e.g. eu-west-1 or us-east-1
• GCP Artifact Registry
  • GCP_PROJECT_ID - ID of your GCP project that hosts of the Artifact Registry
  • GCP_REGION - GCP Region for the Artifact Registry. For example, europe-west4 or us-east1
  • GCP_NEW_REPOSITORY (optional) - Use this environment variable if you want to create a new repository in your Artifact Registry. Note that repository names may only contain lowercase letters, numbers, and hyphens, and must begin with a letter and end with a letter or number.

Authentication

Pushing Docker image to a registry requires authentication via the docker login command. This example contains three different use cases:

• Docker Hub login with credentials
• AWS ECR login with machine identity
• GCP Artifact Registry login with service account 

If you need to connect to another registry, you can use the library steps here as examples to create your own.

Step: docker-image-dockerhub

This step uses the Docker Hub username and password for authentication. You will have to provide them as the environment variables DHUSERNAME and DHPASSWORD, respectively.

You can add them

• under the project setting,
• as an environment variable group in the organization management menu,
• or, separately for each execution.

Remember to make the password a secret! 

Step: docker-image-aws

It is possible to use the machine identity to provide access to AWS ECR. This way you do not need to save the credentials in environment variables. 

We recommend, that you create a separate role for in your AWS account for the environment type that is allowed to push imaged to your ECR, e.g. ValohaiWorkerRoleECR. You will need to add this role in the ValohaiMasterPolicy. 

     {
       "Sid" : "4",
       "Effect" : "Allow",
       "Action" : [
         "iam:PassRole",
         "iam:GetRole"
       ],
       "Resource" : [
          "arn:aws:iam::<ACCOUNT-ID>:role/ValohaiWorkerRole",
          "arn:aws:iam::<ACCOUNT-ID>:role/ValohaiWorkerRoleECR"
       ]
     },

You should also attach the existing ValohaiWorkerPolicy to the new role. In addition, you will need to attach the following policy. Replace the <ACCOUNT-ID> and the <REPOSITORY> with your own values. See the AWS documentation for more information.

• You can also create a new policy, e.g. ValohaiWorkerPolicyECR, that combines the two policies.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListImagesInRepository",
            "Effect": "Allow",
            "Action": [
                "ecr:ListImages"
            ],
            "Resource": "arn:aws:ecr:eu-west-1:<ACCOUNT-ID>:repository/<REPOSITORY>*"
        },
        {
            "Sid": "ManageRepositoryContents",
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:PutImage"
                "ecr:UntagResource",
                "ecr:CreateRepository",
                "ecr:TagResource"
            ],
            "Resource": "arn:aws:ecr:eu-west-1:<ACCOUNT-ID>:repository/<REPOSITORY>*"
        },
        {
            "Effect": "Allow",
            "Action": "ecr:GetAuthorizationToken",
            "Resource": "*"
        }
    ]
}

Finally, inform Valohai Customer Team (support@valohai.com) about the new role name so that they can setup the environment for you. 

Step: docker-image-gcp

In GCP you can use service accounts to push Docker images to the Artifact Registry. You can read more about the permissions from GCP documentation. 

1. Create a new service account under your Google Cloud project that contains the registry.
2. Add Artifact Registry Repository Administrator role so it can push Docker images to your private repository and also create new repositories. 

Using the pushed image

In order to use the pushed image from your private registry in your executions, an admin member has to connect it to your Valohai organization. See the documentation on Private Docker Registries to learn how to do that.