The default work directory within the container is /workspace, which is also the root directory of the Jupyter instance. The /workspace directory is intended to be used for all the important work artifacts. Data within other directories of the server (e.g., /root) might get lost at container restarts.

Token-based Authentication via Jupyter (recommended)

Activate the token-based authentication based on the authentication implementation of Jupyter via the AUTHENTICATE_VIA_JUPYTER variable:

docker run -p 8080:8080 --env AUTHENTICATE_VIA_JUPYTER="mytoken" mltooling/ml-workspace:0.13.2

You can also use <generated> to let Jupyter generate a random token that is printed out on the container logs. A value of true will not set any token but activate that every request to any tool in the workspace will be checked with the Jupyter instance if the user is authenticated. This is used for tools like JupyterHub, which configures its own way of authentication.

Basic Authentication via Nginx

Activate the basic authentication via the WORKSPACE_AUTH_USER and WORKSPACE_AUTH_PASSWORD variable:

docker run -p 8080:8080 --env WORKSPACE_AUTH_USER="user" --env WORKSPACE_AUTH_PASSWORD="pwd" mltooling/ml-workspace:0.13.2

The basic authentication is configured via the nginx proxy and might be more performant compared to the other option since with AUTHENTICATE_VIA_JUPYTER every request to any tool in the workspace will check via the Jupyter instance if the user (based on the request cookies) is authenticated.

When set to true, either the cert.crt and cert.key file must be mounted to /resources/ssl or, if the certificate files do not exist, the container generates self-signed certificates. For example, if the /path/with/certificate/files on the local system contains a valid certificate for the host domain (cert.crt and cert.key file), it can be used from the workspace as shown below:

docker run \
    -p 8080:8080 \
    --env WORKSPACE_SSL_ENABLED="true" \
    -v /path/with/certificate/files:/resources/ssl:ro \
    mltooling/ml-workspace:0.13.2

If you want to host the workspace on a public domain, we recommend to use Let's encrypt to get a trusted certificate for your domain. To use the generated certificate (e.g., via certbot tool) for the workspace, the privkey.pem corresponds to the cert.key file and the fullchain.pem to the cert.crt file.

When you enable SSL support, you must access the workspace over https://, not over plain http://.

For example, the following command restricts the workspace to only use a maximum of 8 CPUs, 16 GB of memory, and 1 GB of shared memory (see Known Issues):

docker run -p 8080:8080 --cpus=8 --memory=16g --shm-size=1G mltooling/ml-workspace:0.13.2

📖 For more options and documentation on resource constraints, please refer to the official docker guide.

The minimal flavor (mltooling/ml-workspace-minimal) is our smallest image that contains most of the tools and features described in the features section without most of the python libraries that are pre-installed in our main image. Any Python library or excluded tool can be installed manually during runtime by the user.

docker run -p 8080:8080 mltooling/ml-workspace-minimal:0.13.2

The R flavor (mltooling/ml-workspace-r) is based on our default workspace image and extends it with the R-interpreter, R-Jupyter kernel, RStudio server (access via Open Tool -> RStudio), and a variety of popular packages from the R ecosystem.

docker run -p 8080:8080 mltooling/ml-workspace-r:0.12.1

The Spark flavor (mltooling/ml-workspace-spark) is based on our R-flavor workspace image and extends it with the Spark runtime, Spark-Jupyter kernel, Zeppelin Notebook (access via Open Tool -> Zeppelin), PySpark, Hadoop, Java Kernel, and a few additional libraries & Jupyter extensions.

docker run -p 8080:8080 mltooling/ml-workspace-spark:0.12.1

Currently, the GPU-flavor only supports CUDA 11.2. Support for other CUDA versions might be added in the future.

The GPU flavor (mltooling/ml-workspace-gpu) is based on our default workspace image and extends it with CUDA 10.1 and GPU-ready versions of various machine learning libraries (e.g., tensorflow, pytorch, cntk, jax). This GPU image has the following additional requirements for the system:

• Nvidia Drivers for the GPUs. Drivers need to be CUDA 11.2 compatible, version >=460.32.03 (📖 Instructions).
• (Docker >= 19.03) Nvidia Container Toolkit (📖 Instructions).

docker run -p 8080:8080 --gpus all mltooling/ml-workspace-gpu:0.13.2

• (Docker < 19.03) Nvidia Docker 2.0 (📖 Instructions).

docker run -p 8080:8080 --runtime nvidia --env NVIDIA_VISIBLE_DEVICES="all" mltooling/ml-workspace-gpu:0.13.2

The GPU flavor also comes with a few additional configuration options, as explained below:

ML Hub makes it easy to set up a multi-user environment on a single server (via Docker) or a cluster (via Kubernetes) and supports a variety of usage scenarios & authentication providers. You can try out ML Hub via:

docker run -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock mltooling/ml-hub:latest

For more information and documentation about ML Hub, please take a look at the Github Site.

1. Start an HTTP server on port 1234 by running this command in a terminal within the workspace: python -m http.server 1234
2. Select Open Tool -> Access Port, input port 1234, and select the Get shareable link option.
3. Click Access, and you will see the content provided by Python's http.server.
4. The opened link can also be shared to other people or called from external applications (e.g., try with Incognito Mode in Chrome).

An SSH connection can be used for tunneling application ports from the remote machine to the local machine, or vice versa. For example, you can expose the workspace internal port 5901 (VNC Server) to the local machine on port 5000 by executing:

ssh -nNT -L 5000:localhost:5901 my-workspace

To expose an application port from your local machine to a workspace, use the -R option (instead of -L).

After the tunnel is established, you can use your favorite VNC viewer on your local machine and connect to vnc://localhost:5000 (default password: vncpassword). To make the tunnel connection more resistant and reliable, we recommend to use autossh to automatically restart SSH tunnels in the case that the connection dies:

autossh -M 0 -f -nNT -L 5000:localhost:5901 my-workspace

Port tunneling is quite useful when you have started any server-based tool within the workspace that you like to make accessible for another machine. In its default setting, the workspace has a variety of tools already running on different ports, such as:

• 8080: Main workspace port with access to all integrated tools.
• 8090: Jupyter server.
• 8054: VS Code server.
• 5901: VNC server.
• 22: SSH server.

You can find port information on all the tools in the supervisor configuration.

📖 For more information about port tunneling/forwarding, we recommend this guide.

SCP allows files and directories to be securely copied to, from, or between different machines via SSH connections. For example, to copy a local file (./local-file.txt) into the /workspace folder inside the workspace, execute:

scp ./local-file.txt my-workspace:/workspace

To copy the /workspace directory from my-workspace to the working directory of the local machine, execute:

scp -r my-workspace:/workspace .

📖 For more information about scp, we recommend this guide.

Rsync is a utility for efficiently transferring and synchronizing files between different machines (e.g., via SSH connections) by comparing the modification times and sizes of files. The rsync command will determine which files need to be updated each time it is run, which is far more efficient and convenient than using something like scp or sftp. For example, to sync all content of a local folder (./local-project-folder/) into the /workspace/remote-project-folder/ folder inside the workspace, execute:

rsync -rlptzvP --delete --exclude=".git" "./local-project-folder/" "my-workspace:/workspace/remote-project-folder/"

If you have some changes inside the folder on the workspace, you can sync those changes back to the local folder by changing the source and destination arguments:

rsync -rlptzvP --delete --exclude=".git" "my-workspace:/workspace/remote-project-folder/" "./local-project-folder/"

You can rerun these commands each time you want to synchronize the latest copy of your files. Rsync will make sure that only updates will be transferred.

📖 You can find more information about rsync on this man page.

Besides copying and syncing data, an SSH connection can also be used to mount directories from a remote machine into the local filesystem via SSHFS.
For example, to mount the /workspace directory of my-workspace into a local path (e.g. /local/folder/path), execute:

sshfs -o reconnect my-workspace:/workspace /local/folder/path

Once the remote directory is mounted, you can interact with the remote file system the same way as with any local directory and file.

📖 For more information about sshfs, we recommend this guide.

The workspace can be added to a Jupyter instance as a remote kernel by using the remote_ikernel tool. If you have installed remote_ikernel (pip install remote_ikernel) on your local machine, the SSH setup script of the workspace will automatically offer you the option to setup a remote kernel connection.

When running kernels on remote machines, the notebooks themselves will be saved onto the local filesystem, but the kernel will only have access to the filesystem of the remote machine running the kernel. If you need to sync data, you can make use of rsync, scp, or sshfs as explained in the SSH Access section.

In case you want to manually setup and manage remote kernels, use the remote_ikernel command-line tool, as shown below:

# Change my-workspace with the name of a workspace SSH connection
remote_ikernel manage --add \
    --interface=ssh \
    --kernel_cmd="ipython kernel -f {connection_file}" \
    --name="ml-server (Python)" \
    --host="my-workspace"

You can use the remote_ikernel command line functionality to list (remote_ikernel manage --show) or delete (remote_ikernel manage --delete <REMOTE_KERNEL_NAME>) remote kernel connections.

The Visual Studio Code Remote - SSH extension allows you to open a remote folder on any remote machine with SSH access and work with it just as you would if the folder were on your own machine. Once connected to a remote machine, you can interact with files and folders anywhere on the remote filesystem and take full advantage of VS Code's feature set (IntelliSense, debugging, and extension support). The discovers and works out-of-the-box with passwordless SSH connections as configured by the workspace SSH setup script. To enable your local VS Code application to connect to a workspace:

1. Install Remote - SSH extension inside your local VS Code.
2. Run the SSH setup script of a selected workspace as explained in the SSH Access section.
3. Open the Remote-SSH panel in your local VS Code. All configured SSH connections should be automatically discovered. Just select any configured workspace connection you like to connect to as shown below:

📖 You can find additional features and information about the Remote SSH extension in this guide.

To run Python code as a job, you need to provide a path or URL to a code directory (or script) via EXECUTE_CODE. The code can be either already mounted into the workspace container or downloaded from a version control system (e.g., git or svn) as described in the following sections. The selected code path needs to be python executable. In case the selected code is a directory (e.g., whenever you download the code from a VCS) you need to put a __main__.py file at the root of this directory. The __main__.py needs to contain the code that starts your job.

Run code from version control system

You can execute code directly from Git, Mercurial, Subversion, or Bazaar by using the pip-vcs format as described in this guide. For example, to execute code from a subdirectory of a git repository, just run:

docker run --env EXECUTE_CODE="git+https://github.com/ml-tooling/ml-workspace.git#subdirectory=resources/tests/ml-job" mltooling/ml-workspace:0.13.2

📖 For additional information on how to specify branches, commits, or tags please refer to this guide.

Run code mounted into the workspace

In the following example, we mount and execute the current working directory (expected to contain our code) into the /workspace/ml-job/ directory of the workspace:

docker run -v "${PWD}:/workspace/ml-job/" --env EXECUTE_CODE="/workspace/ml-job/" mltooling/ml-workspace:0.13.2

Install Dependencies

In the case that the pre-installed workspace libraries are not compatible with your code, you can install or change dependencies by just adding one or multiple of the following files to your code directory:

• requirements.txt: pip requirements format for pip-installable dependencies.
• environment.yml: conda environment file to create a separate Python environment.
• setup.sh: A shell script executed via /bin/bash.

The execution order is 1. environment.yml -> 2. setup.sh -> 3. requirements.txt

Test job in interactive mode

You can test your job code within the workspace (started normally with interactive tools) by executing the following python script:

python /resources/scripts/execute_code.py /path/to/your/job

Build a custom job image

It is also possible to embed your code directly into a custom job image, as shown below:

FROM mltooling/ml-workspace:0.13.2

Add job code to image
COPY ml-job /workspace/ml-job
ENV EXECUTE_CODE=/workspace/ml-job
Install requirements only
RUN python /resources/scripts/execute_code.py --requirements-only
Execute only the code at container startup
CMD ["python", "/resources/docker-entrypoint.py", "--code-only"]

For example, to install the Apache Zeppelin notebook server, simply execute:

/resources/tools/zeppelin.sh --port=1234

After installation, refresh the Jupyter website and the Zeppelin tool will be available under Open Tool -> Zeppelin. Other tools might only be available within the Desktop VNC (e.g., atom or pycharm) or do not provide any UI (e.g., starspace, docker-client).

The workspace can be extended in many ways at runtime, as explained here. However, if you like to customize the workspace image with your own software or configuration, you can do that via a Dockerfile as shown below:

# Extend from any of the workspace versions/flavors
FROM mltooling/ml-workspace:0.13.2

Run you customizations, e.g.
RUN 
    # Install r-runtime, r-kernel, and r-studio web server from provided install scripts
    /bin/bash $RESOURCES_PATH/tools/r-runtime.sh --install && 
    /bin/bash $RESOURCES_PATH/tools/r-studio-server.sh --install && 
    # Cleanup Layer - removes unneccessary cache files
    clean-layer.sh

Finally, use docker build to build your customized Docker image.

📖 For a more comprehensive Dockerfile example, take a look at the Dockerfile of the R-flavor.

To update a running workspace instance to a more recent version, the running Docker container needs to be replaced with a new container based on the updated workspace image.

All data within the workspace that is not persisted to a mounted volume will be lost during this update process. As mentioned in the persist data section, a volume is expected to be mounted into the /workspace folder. All tools within the workspace are configured to make use of the /workspace folder as the root directory for all source code and data artifacts. During an update, data within other directories will be removed, including installed/updated libraries or certain machine configurations. We have integrated a backup and restore feature (CONFIG_BACKUP_ENABLED) for various selected configuration files/folders, such as the user's Jupyter/VS-Code configuration, ~/.gitconfig, and ~/.ssh.

docker run -d \
    -p 8080:8080 \
    --name "ml-workspace" \
    -v "/path/on/host:/workspace" \
    --env AUTHENTICATE_VIA_JUPYTER="mytoken" \
    --restart always \
    mltooling/ml-workspace:0.8.7

If you want to directly connect to the workspace via a VNC client (not using the noVNC webapp), you might be interested in changing certain VNC server configurations. To configure the VNC server, you can provide/overwrite the following environment variables at container start (via docker run option: --env):

Unfortunately, we currently do not support using a non-root user within the workspace. We plan to provide this capability and already started with some refactoring to allow this configuration. However, this still requires a lot more work, refactoring, and testing from our side.

Using root-user (or users with sudo permission) within containers is generally not recommended since, in case of system/kernel vulnerabilities, a user might be able to break out of the container and be able to access the host system. Since it is not very common to have such problematic kernel vulnerabilities, the risk of a severe attack is quite minimal. As explained in the official Docker documentation, containers (even with root users) are generally quite secure in preventing a breakout to the host. And compared to many other container use-cases, we actually want to provide the flexibility to the user to have control and system-level installation permissions within the workspace container.

The workspace comes preinstalled with various common tools to create isolated Python environments (virtual environments). The following sections provide a quick-intro on how to use these tools within the workspace. You can find information on when to use which tool here. Please refer to the documentation of the given tool for additional usage information.

venv (recommended):

To create a virtual environment via venv, execute the following commands:

# Create environment in the working directory
python -m venv my-venv
# Activate environment in shell
source ./my-venv/bin/activate
# Optional: Create Jupyter kernel for this environment
pip install ipykernel
python -m ipykernel install --user --name=my-venv --display-name="my-venv ($(python --version))"
# Optional: Close enviornment session
deactivate

pipenv (recommended):

To create a virtual environment via pipenv, execute the following commands:

# Create environment in the working directory
pipenv install
# Activate environment session in shell
pipenv shell
# Optional: Create Jupyter kernel for this environment
pipenv install ipykernel
python -m ipykernel install --user --name=my-pipenv --display-name="my-pipenv ($(python --version))"
# Optional: Close environment session
exit

virtualenv:

To create a virtual environment via virtualenv, execute the following commands:

# Create environment in the working directory
virtualenv my-virtualenv
# Activate environment session in shell
source ./my-virtualenv/bin/activate
# Optional: Create Jupyter kernel for this environment
pip install ipykernel
python -m ipykernel install --user --name=my-virtualenv --display-name="my-virtualenv ($(python --version))"
# Optional: Close environment session
deactivate

conda:

To create a virtual environment via conda, execute the following commands:

# Create environment (globally)
conda create -n my-conda-env
# Activate environment session in shell
conda activate my-conda-env
# Optional: Create Jupyter kernel for this environment
python -m ipykernel install --user --name=my-conda-env --display-name="my-conda-env ($(python --version))"
# Optional: Close environment session
conda deactivate

Tip: Shell Commands in Jupyter Notebooks:

If you install and use a virtual environment via a dedicated Jupyter Kernel and use shell commands within Jupyter (e.g. !pip install matplotlib), the wrong python/pip version will be used. To use the python/pip version of the selected kernel, do the following instead:

import sys
!{sys.executable} -m pip install matplotlib

The workspace provides three easy options to install different Python versions alongside the main Python instance: pyenv, pipenv (recommended), conda.

pipenv (recommended):

To install a different python version (e.g. 3.7.8) within the workspace via pipenv, execute the following commands:

# Install python vers
pipenv install --python=3.7.8
# Activate environment session in shell
pipenv shell
# Check python installation
python --version
# Optional: Create Jupyter kernel for this environment
pipenv install ipykernel
python -m ipykernel install --user --name=my-pipenv --display-name="my-pipenv ($(python --version))"
# Optional: Close environment session
exit

pyenv:

To install a different python version (e.g. 3.7.8) within the workspace via pyenv, execute the following commands:

# Install python version
pyenv install 3.7.8
# Make globally accessible
pyenv global 3.7.8
# Activate python version in shell
pyenv shell 3.7.8
# Check python installation
python3.7 --version
# Optional: Create Jupyter kernel for this python version
python3.7 -m pip install ipykernel
python3.7 -m ipykernel install --user --name=my-pyenv-3.7.8 --display-name="my-pyenv (Python 3.7.8)"

conda:

To install a different python version (e.g. 3.7.8) within the workspace via conda, execute the following commands:

# Create environment with python version
conda create -n my-conda-3.7 python=3.7.8
# Activate environment session in shell
conda activate my-conda-3.7
# Check python installation
python --version
# Optional: Create Jupyter kernel for this python version
pip install ipykernel
python -m ipykernel install --user --name=my-conda-3.7 --display-name="my-conda ($(python --version))"
# Optional: Close environment session
conda deactivate

Tip: Shell Commands in Jupyter Notebooks:

If you install and use another Python version via a dedicated Jupyter Kernel and use shell commands within Jupyter (e.g. !pip install matplotlib), the wrong python/pip version will be used. To use the python/pip version of the selected kernel, do the following instead:

import sys
!{sys.executable} -m pip install matplotlib

Certain desktop tools (e.g., recent versions of Firefox) or libraries (e.g., Pytorch - see Issues: 1, 2) might crash if the shared memory size (/dev/shm) is too small. The default shared memory size of Docker is 64MB, which might not be enough for a few tools. You can provide a higher shared memory size via the shm-size docker run option:

docker run --shm-size=2G mltooling/ml-workspace:0.13.2

In general, the performance of running code within Docker is nearly identical compared to running it directly on the machine. However, in case you have limited the container's CPU quota (as explained in this section), the container can still see the full count of CPU cores available on the machine and there is no technical way to prevent this. Many libraries and tools will use the full CPU count (e.g., via os.cpu_count()) to set the number of threads used for multiprocessing/-threading. This might cause the program to start more threads/processes than it can efficiently handle with the available CPU quota, which can tremendously slow down the overall performance. Therefore, it is important to set the available CPU count or the maximum number of threads explicitly to the configured CPU quota. The workspace provides capabilities to detect the number of available CPUs automatically, which are used to configure a variety of common libraries via environment variables such as OMP_NUM_THREADS or MKL_NUM_THREADS. It is also possible to explicitly set the number of available CPUs at container startup via the MAX_NUM_THREADS environment variable (see configuration section). The same environment variable can also be used to get the number of available CPUs at runtime.

Even though the automatic configuration capabilities of the workspace will fix a variety of inefficiencies, we still recommend configuring the number of available CPUs with all libraries explicitly. For example:

import os
MAX_NUM_THREADS = int(os.getenv("MAX_NUM_THREADS"))

Set in pytorch
import torch
torch.set_num_threads(MAX_NUM_THREADS)
Set in tensorflow
import tensorflow as tf
config = tf.ConfigProto(
    device_count={"CPU": MAX_NUM_THREADS},
    inter_op_parallelism_threads=MAX_NUM_THREADS,
    intra_op_parallelism_threads=MAX_NUM_THREADS,
)
tf_session = tf.Session(config=config)
Set session for keras
import keras.backend as K
K.set_session(tf_session)
Set in sklearn estimator
from sklearn.linear_model import LogisticRegression
LogisticRegression(n_jobs=MAX_NUM_THREADS).fit(X, y)
Set for multiprocessing pool
from multiprocessing import Pool
with Pool(MAX_NUM_THREADS) as pool:
    results = pool.map(lst)

If you encounter the following error within the container logs when starting the workspace, it will most likely not be possible to run the workspace on your hardware:

exited: nginx (terminated by SIGILL (core dumped); not expected)

The OpenResty/Nginx binary package used within the workspace requires to run on a CPU with SSE4.2 support (see this issue). Unfortunately, some older CPUs do not have support for SSE4.2 and, therefore, will not be able to run the workspace container. On Linux, you can check if your CPU supports SSE4.2 when looking into the cat /proc/cpuinfo flags section. If you encounter this problem, feel free to notify us by commenting on the following issue: #30.