Skip to main content

Bring your existing projects to Valohai

drazen
  • Updated
git repository
Connect to a Git repository		 docker
Choose a Docker image		 yaml configuration file
Add a valohai.yaml configuration		 data
Use Valohai data inputs/outputs		 parameters
Define parameters in valohai.yaml		 metadata
Collect key metrics in JSON

This guide assumes that your organization has the environments and data stores configured already as a part of the Valohai installation.

This guide assumes your project is written in Python and that you want to use the valohai-utils Python helper library in your projects (recommended).

If you're not using Python, or not planning to use valohai-utils, then we suggest you check out:

Connect your Git repository to a Valohai project

Start by going to the Valohai application (app.valohai.com) and creating a new project.

The owner of the project should be your organization or team so that you can access your organization's cloud resources like virtual machines, data stores, and private Docker registries.

After you’ve created the project you’ll need to connect it to a Git repository through the Repository tab in your project’s settings.

Install Valohai tools

Valohai offers command-line tools and a Python helper library to make it easier to interact with the platform.

Start by installing the Valohai tools, logging in, and linking your local directory to a Valohai project.

pip3 install valohai-utils valohai-cli
vh login

# Navigate to your project
cd myproject

# Links your current working directory to a existing Valohai project
vh project link

Choose a Docker image

Each Valohai execution is run inside a Docker container. This makes it possible to execute any code from C to Python as long as it can run inside the chosen container.

You can use any Docker images for your Valohai jobs, either from a public or a private Docker registry.

Here are the most common Docker images currently used on the platform:

tensorflow/tensorflow:-gpu                                  # e.g. 2.6.1-gpu, for GPU support
tensorflow/tensorflow:                                      # e.g. 2.6.1, for CPU only
pytorchlightning/pytorch_lightning:latest-py-torch # e.g. py3.6-torch1.6
pytorch/pytorch:                                            # e.g. 1.3-cuda10.1-cudnn7-runtime
python:                                                     # e.g. 3.8.0
r-base:                                                     # e.g. 3.6.1
julia:                                                      # e.g. 1.3.0
valohai/prophet                                                    # Valohai hosted image with Prophet
valohai/sklearn:1.0                                                  # Valohai hosted image with sklean
valohai/xgboost:1.4.2                                                 # Valohai hosted image with xgboost

You can also install any additional libraries inside the execution with for example apt-get install or pip install -r requirements.txt.

Having a custom Docker image allows you to bake in all the libraries and other dependencies that your machine learning code has. This way you don’t need to spend time running additional installation commands at the beginning of every execution to get everything you need.

Valohai will download and cache the Docker images on the worker virtual machine, so they don’t have to be redownloaded for every job.

Next

To get the most out of Valohai features you should integrate with Valohai input/output data, parameters, and metadata system. Follow the guides below for details:

Topic

Why

Guide

Configuration file

  • Valohai expects that you have a valohai.yaml configuration file in your repository. The file defines what kind of jobs can be executed inside your project, and the different properties of each job type (= Step).

Add a valohai.yaml configuration file

Files

  • Data scientists don’t need to worry about authenticating with your cloud object store, downloading/uploading files, and caching data.

  • In Valohai you always treat files from an object store as “local files”. Valohai downloads your data to a local directory in /valohai/inputs/. It will pick up any files from /valohai/outputs/ and upload them to your private cloud storage.

Input and output data

Parameters

  • Keep track of parameters used in your executions, and rerun any job with the same or changed parameter values.

  • Easily run parameter sweeps or hyperparameter optimization by using Valohai Tasks.

Define Valohai parameters

Metrics

  • Easily keep track, compare, and sort your jobs based on custom metrics that you collect.

  • Visualize and compare execution metrics in a time series graph, a scatter plot, or a confusion matrix.

Collect metadata

Related articles

Comments

0 comments

Please sign in to leave a comment.