Using an existing Kubernetes cluster as a deployment target for real-time inference

Valohai can push deployments to an existing Kubernetes cluster.

Valohai uses standard Kubernetes APIs to communicate with your Kubernetes cluster and app.valohai.com (34.248.245.191) should be able to access your clusters API Server over HTTPS.

You cluster can be configured to serve only private deployment endpoints.

• Install ingress-nginx on the cluster

  • https://kubernetes.github.io/ingress-nginx/deploy/

• Get the external IP of your ingress-nginx. You’ll need to share this with Valohai.

  kubectl -n ingress-nginx get service/ingress-nginx-controller

• Create a Kubernetes service account that Valohai will use

  kubectl create serviceaccount valohai-deployment

• Find the token name (one secret token should be generated automatically). You’ll need to provide this token back to Valohai.

  kubectl get serviceaccounts valohai-deployment -o json kubectl get secret valohai-deployment-token- -o json

• Setup the valohai-deployment-role in Kuberenetes. 

  • Create a new file valohai-deployment-role.yml with the contents specified below.

  • If you need to limit access to a certain namespace, you can add namespace: <NAMESPACE> under metadata. 

  apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: valohai-deployment-role
  rules:
    - apiGroups: [""]
      resources: ["events", "namespaces"]
      verbs: ["get", "list", "watch"]
    - apiGroups: [""]
      resources: ["pods", "pods/log", "services"]
      verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
    - apiGroups: ["apps", "extensions"]
      resources: ["deployments", "deployments/rollback", "deployments/scale"]
      verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
    - apiGroups: ["extensions", "networking.k8s.io"]
      resources: ["ingresses"]
      verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  

  • Apply the role with kubectl apply -f valohai-deployment-role.yml

  • Create a rolebinding

  kubectl create rolebinding valohai-deployment-binding \
      --role=valohai-deployment-role \
      --serviceaccount=<namespace>:valohai-deployment
  

Make sure your cluster’s nodes can pull from the repository that Valohai is pushing images to.

User Account

This user is required so Valohai can access the cluster and deploy new images to your ECR.

• Create a user valohai-eks-user.

  • Enable Programmatic access and Console access

  Attach the following existing policies:

  • AmazonEC2ContainerRegistryFullAccess

  • AmazonEKSServicePolicy

• Click on Create policy to open a new tab. Describe the new policy with the JSON below.

   {
      "Version": "2012-10-17",
    	"Statement": [
        {
            "Sid": "1",
            "Effect": "Allow",
            "Action": "eks:ListClusters",
            "Resource": "*"
        }
    ]
  }

• Name the policy VH_EKS_USER and create it.

• Back in your Add user tab click on the refresh button and select the VH_EKS_USER policy.

• Store the access key & secret in a safe place.

Other

You can use standard Docker login (username/password) credentials when pushing to Azure Container Registry, GitLab, Artifactory, Docker Hub, and others.

Make sure you create a seperate account for Valohai to be able to push to your repository.