Launch a execution when a new commit is pushed to a GitHub repository

GitHub Actions allow you to automate development workflows right from your repository. In this tutorial, we’ll use GitHub Actions to trigger a new execution on Valohai every time a new push is made to our repository.

Prerequisites:

• Existing Valohai project with a valohai.yaml file and at least one step. You can find a sample valohai.yaml and train.py from the sample repository of this tutorial.
• Your project is connected to a GitHub repository (either public or private)

Generate a Valohai API Token

You'll need to generate a Valohai API key to authorize your requests with your Valohai projects.

• Go to https://app.valohai.com
• Click on the "Hi, <username>!" on the top-right corner
• Go to My Profile -> Authentication
• Click on Manage Tokens and scroll to the bottom of the page to generate a new token

Make a note of the token that pops up on the top of your page. It will be shown only once.

Add a GitHub Repository Secret

Save your Valohai API token as a secret in your repository to access it from your GitHub Actions.

• Go to your GitHub Project
• Open Settings -> Secrets -> Actions
• Create a new Repository Secret
  • Name: VH_API_TOKEN
  • Value: paste your Valohai API Key

Create your GitHub Action workflow

Workflows are used to define an automated process for your repository. In our case we’ll define a workflow that triggers a new action every time there is a push to the repository.

A workflow needs to know when it should be triggered, what are the jobs to run and in which environment should it run these jobs.

• In your existing project, create a new folder .github/
• Add a new folder to keep our workflows .github/workflows/
• Create a new file called .github/workflows/main.yml and define the workflow as:

name: Workflow for Create Valohai execution on push
on: push
jobs:
  build:
      name: Create Valohai execution on push
      runs-on: ubuntu-latest
      env:
        VH_API_TOKEN: ${{ secrets.VH_API_TOKEN }}
      steps:
      - uses: actions/checkout@v1
      - uses: ./valohai-execution-action

on:
  push:
    branches:
      - master
      - main

Define the action that should be triggered

GitHub Actions can run directly on the machine (JavaScript) on in a Docker container.

Our action will run a Python script inside a Docker container. That script will use the Valohai APIs to trigger a new Valohai execution based on the new commit.

• Start by creating a new folder in the root of your directory valohai-execution-action/.
• Create a new file valohai-execution-action/runExecution.py. In there create a new call to the Valohai APIs.
  • You’ll need an authentication token and know your project ID to continue. You can find your project’s ID under its Settings-tab.

import requests
import json
import os

# Authenticate yourself with the token.
# Remember to follow your organization's security standards when handling the token.   
auth_token = os.getenv('VH_API_TOKEN')
headers = {'authorization': 'token %s' % auth_token}
project_id = 'your-project-id'
step_name = 'your-step-name-from-valohai.yaml'

# Fetch all new changes from the repository
# https://app.valohai.com/api/docs/#projects-fetch
# This will fetch changes from all the branches that you've defined on the Project->Settings->Repository tab
fetchResponse = requests.post(('https://app.valohai.com/api/v0/projects/{0}/fetch/').format(project_id), data={'id': project_id}, headers=headers)
fetchResponse.raise_for_status()

# Define the payload for a new execution
# https://app.valohai.com/api/docs/#executions-create
#
# GitHub Actions creates an environment variable on the Docker container
# Called GITHUB_SHA that stores the identifier of the commit that was created
new_exec_payload = {
   "project": project_id,
   "commit": os.getenv('GITHUB_SHA'),
   "step": step_name
}

# Send a POST request to create a new execution
createExecutionResponse = requests.post('https://app.valohai.com/api/v0/executions/', json=new_exec_payload, headers=headers)
createExecutionResponse.raise_for_status()

# Print the response you've received back
print('# API Response:\n')
print(json.dumps(createExecutionResponse.json(), indent=4))

• Next, create a valohai-execution-action/Dockerfile that will be used to build your container.

# We'll use a slim python image as a base
FROM python:3.8.2-slim-buster

# Our code will need requests, so we can install them on the image with pip
RUN pip install requests

# Add our file and run it
ADD runExecution.py /runExecution.py
CMD ["/runExecution.py"]
ENTRYPOINT ["python"]

• Finally, we’ll need to create valohai-execution-action/action.yml to describe our action for GitHub.

name: "Create Valohai execution on push"
description: "Run a new execution on app.valohai.com on each new push to repository"
author: "your-email"

runs:
  using: "docker"
  image: "Dockerfile"

Push a new commit

Ta-da! That’s it! Now you can commit your code and the push changes.

• git add .github/ *
• git commit -m "added github action to trigger vh executions"`
• git push

Navigate to your GitHub repository’s Actions tab to see the new action…in action.

You can test it out by making some changes to your step’s code (e.g. train.py) and push the changes to GitHub. You should see the Action trigger and create a new execution on Valohai with that new commit.