Yes.
- In hybrid installations (using app.valohai.com) you'll need to setup a small EC2 instance that can route requests from app.valohai.com to the Redis cluster, as by default the AWS ElastiCache is not accessible from outside of it's VPC.
- In self-hosted installations you'll need to make sure the Valohai Roi and workers are in the same VPC as the Redis cluster.
For additional details see: Authorize access to the cluster in the AWS docs.
Allow app.valohai.com to access your Amazon ElastiCache
You'll need to setup a tiny EC2 instance that will accept connections from app.valohai.com, handle TLS-termination, and direct the request the requests to the Amazon ElastiCache in the private network.
Create an EC2 instance
Start by creating a new EC2 instance in the same VPC where the Amazon ElastiCache instance is:
- Name: valohai-queue-redirector
- AMI: Ubuntu Server 22.04 LTS
- Instance type: t2.micro
- Key Pair: select an existing key pair, or generate a new one
- Network settings:
- VPC: Choose the same VPC where your Amazon ElastiCache is
- Security Group: Create a security group
- Name: valohai-queue-redirect-sg
- Description: Allow Lets Encrypt certificate generation and Redis connections from app.valohai.com
- Inbound Security Group rules:
- Port 22, Source Type: Choose My IP as the Source Type
- Add two new rules:
- Rule #1
- Type: HTTP
- Source Type: Anywhere
- Description: Used by Let's Encrypt certification verification (http challenge)
- Rule #2
- Type: Custom TCP
- Port Range: 63790
- Source Type: Custom
- Source: 34.248.245.191/32
- That's the IP of app.valohai.com
- Description: Redis access from app.valohai.com
- Rule #1
- Configure storage:
- 1 x 16 GiB gp2
Click launch instance.
Assign an Elastic IP
Next go the Elastic IP page on EC2 and click to Allocate Elastic IP address with the default settings.
After the IP is created you can click Actions -> Associate Elastic IP Address and choose the valohai-redis-redirector instance from the list.
Allow valohai-queue-redirect-sg to access Redis
Edit the Security Group rules of your Amazon ElastiCache to allow a new inbound rule:
- Type: Custom TCP
- Port Range: 6379
- Source: valohai-queue-redirect-sg
- Description: Allow connections from valohai-queue-redirector EC2 instance.
Make sure you also have a rule with the port but the source as valohai-queue-redirect-sg.
Setup Traefik on the EC2 instance
Now ssh into the machine using the key you specified when creating the instance:
ssh -i <path-to-key.pem> ubuntu@<public-IP-of-EC2-instance>
Follow the instructions on docs.docker.com to install Docker on your EC2 instance using the repository.
Next we'll use docker-compose to configure the Traefik image.
docker-compose.yml
Create a new file called docker-compose.yml
with the following configuration.
- Update the
YOUR-EMAIL
with your own email.
services:
traefik:
image: "traefik:v2.7"
container_name: "traefik"
command:
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--providers.file.filename=/etc/traefik/rules.yml"
- "--entrypoints.web.address=:80"
- "--entrypoints.redis.address=:63790"
- "--certificatesresolvers.certresolver.acme.httpchallenge=true"
- "--certificatesresolvers.certresolver.acme.httpchallenge.entrypoint=web"
- "--certificatesresolvers.certresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
- "--certificatesresolvers.certresolver.acme.email=YOUR-EMAIL"
- "--certificatesresolvers.certresolver.acme.storage=/letsencrypt/acme.json"
ports:
- "80:80"
- "63790:63790"
volumes:
- "./letsencrypt:/letsencrypt"
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./rules.yml:/etc/traefik/rules.yml"
rules.yaml
Next create a file called rules.yml
:
- Update the
YOUR-QUEUE-ADDRESS
with the address you receive from your Valohai contact - Update the
REDIS-PRIMARY-ENDPOINT
with the Primary endpoint of your Redis cluster. You can find this in your AWS Console under the Cluster details page in Amazon ElastiCache.
tcp:
routers:
redis-route:
rule: "HostSNI(`YOUR-QUEUE-ADDRESS`)"
entrypoints:
- redis
service: redis-server
tls:
passthrough: true
certresolver: certresolver
services:
redis-server:
loadBalancer:
servers:
- address: "REDIS-PRIMARY-ENDPOINT"
Now launch Traefik with docker-compose and make sure everything launches without errors.
sudo docker compose up
You can then exit with CMD+C / Ctrl+C.
Launch on startup
We'll want to redirector to launch on machine startup, in case the machine needs to be restarted at some point.
Create a new file at:
sudo nano /etc/systemd/system/queue-redirector.service
and paste in the following:
[Unit]
Description=Valohai Queue Redirector
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/home/ubuntu/
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
Enable the service with
sudo systemctl daemon-reload
sudo systemctl enable queue-redirector
Finally, you'll need to send Valohai the Public IP of your EC2 instance.
Comments
0 comments
Please sign in to leave a comment.