Attach a debugger (PyCharm)

In this guide, we’ll look at attaching the PyCharm remote debugger to a Valohai execution.

Sample code

For this example we’ll use the following simple example code:

import numpy as np
import time

for x in range(1,20) :
    print(f"Doing computation {x}")
    data = np.random.random((50,50))
    sum = np.sum(data)
    time.sleep(2)

And the corresponding valohai.yaml file:

- step:
  name: train
  image: python:3.9
  command:
    - pip install numpy
    - python train.py

Create a new Debug Configuration in PyCharm

Create a new Python Debug Server in PyCharm. The IDE host name should be localhost and the port should be the port that you want to have listening to incoming debug connections on your local machine (here 1234).

You can also now start the created Debug Server and leave it to wait for the process connection.

Use pydevd_pycharm

Import pydevd_pycharm and set it to wait for connection with the debugger.

import pydevd_pycharm
import numpy as np
import time

# The script is halted here, until a debugger is attached
connected = False

while not connected:
    try:
        # The arbitrary container port 1234 on the worker machine is set to connect with the debugger.
        # Note that this is not the same port you specified during step 1 as that is on your local machine.
        pydevd_pycharm.settrace('localhost', port=1234, stdoutToServer=True, stderrToServer=True)
        connected = True
    except Exception:
        print("Waiting for connection...")
        time.sleep(10)

for x in range(1,20) :
    print(f"Doing computation {x}")
    data = np.random.random((50,50))
    sum = np.sum(data)
    time.sleep(2)

Now update the valohai.yaml file to install pydevd_pycharm in your Valohai execution. You can copy the correct PyCharm version for example from the Run Configuration section.

- step:
  name: train
  image: python:3.9
  command:
    - pip install numpy pydevd_pycharm~=<PYCHARM-VERSION>
    - python train.py

Commit your changes and push them to your Git repository.

Then launch a new Valohai execution with SSH debugging enabled. Follow the Remote access (SSH) how-to guide for detailed instructions.

Connect from PyCharm to a remote execution

Since pydevd_pycharm does not have a listen mode, you will need to establish a reverse SSH tunnel between your local machine and the Valohai worker instance. You’ll get the IP of the worker machine from the Valohai execution logs.

ssh -i <PATH-TO-YOUR-PRIVATE-SSH-KEY> <IP-FROM-VALOHAI> -p 2222 -R 1234:localhost:1234 -t /bin/bash

You might see a prompt saying that the file path on the worker machine cannot be found in the project. This happens because the path mapping was not defined when creating the debugging configuration. You can just click on the “Auto detect” to set the mapping.

Finally, you should see a message in the Debugger console saying that the connection was established.

The debugger will wait for you to tell it to continue with the execution by clicking on the green arrow. When you hit a breakpoint, you’ll also be able to see your local variables and edit them on the fly.