This post enables using Tensorboard within the JupyterHub + K8s ecosystem. First, access to the proxy pod is gained on the primary node by setting up a NodePort. Next, a python script is setup to run as a service that reads and updates the proxy routing table on a timed interval. The routing table gets updated with new routes such as /user/xxx/tb/ that point to standard Tensorboard ports on each of the JupyterHub single user servers. In addition, a proxy is setup on the single user server that corrects the incoming path requests to work with Tensorboard. Lastly, Tensorboard is run on the single user server and is bound to the incoming traffic.
Setup NodePort
In order for an update script to access the routing table a NodePort is setup. This points traffic directed at our specified port to the proxy api (configurable-http-proxy).
apiVersion: v1
kind: Service
metadata:
namespace: jhub
name: proxy-api-nodeport
spec:
type: NodePort
ports:
- port: 8001
nodePort: 31477
targetPort: 8001
selector:
app: jupyterhub
component: proxy
release: [release name]
Code language: YAML (yaml)Apply the Nodeport changes to the cluster with the following command.
kubectl create -f file_name.yamlCode language: CSS (css)Routing Table Update Script (tensorboard-routing.py)
Setup a python virtual environment that has the requests library installed. The following script can then get an authorization token to communicate with the proxy RESTful api over the previously setup NodePort. Every 30 seconds the routing table is read and updated. This creates a new /user/xxx/tb600x/ path for each active single user server. The path points to ports 6006, 6007, and 6008.
import requests
import subprocess
import time
node_port = 31477
api_url = 'http://localhost:' + \
str(node_port) + '/api/routes'
token_descriptor = "hub.config." + \
"ConfigurableHTTPProxy.auth_token"
def get_token():
cmd = ["kubectl", "get", "secret", "hub",
"-n", "jhub", "-o=json"]
p1 = subprocess.run(cmd, stdout=subprocess.PIPE)
cmd = ["jq", "-r", ".[\"data\"] | " +
".[\"" + token_descriptor + "\"]"]
p2 = subprocess.run(cmd, stdout=subprocess.PIPE,
input=p1.stdout)
cmd = ["base64", "--decode"]
p3 = subprocess.run(cmd, stdout=subprocess.PIPE,
input=p2.stdout)
return p3.stdout.decode()
token = ""
while len(token) == 0:
token = get_token()
time.sleep(10)
print('token:', token)
def get_routes():
r = requests.get(api_url,
headers={
'Authorization': 'token %s' % token,
}
)
r.raise_for_status()
routes = r.json()
return routes
def post_route(token=None, user=None, target=None, path=None):
res = requests.post(
api_url + '/user/' + user + '/' + path + '/',
headers={
'Authorization': 'token %s' % token
},
json={
'user': user,
'target': target,
'jupyterHub': True
}
)
print('route post:', res)
while True:
try:
routes = get_routes()
except:
print('failed to get routes')
continue
try:
for key in routes.keys():
items = key.split('/')
tb_path = len(items) > 3 and 'tb' in items[3]
if items[1] == 'user' and not tb_path:
user = routes[key]['user']
target = 'http:' + \
routes[key]['target'].split(':')[1] + ':'
post_route(token, user, target + '6116', 'tb')
post_route(token, user, target + '6117', 'tb6007')
post_route(token, user, target + '6118', 'tb6008')
except:
print('failed to update routes')
time.sleep(30)
Code language: Python (python)This script can be executed manually with the following call.
Code language: Bash (bash)python tensorboard-routing.py
Alternatively, the script can automatically run after reboot using a Linux service. The following contents should be placed in a file named /etc/systemd/system/tensorboard-routing.service.
[Unit]
Description=Service to run a routing table updater for Tensorboard support in JupyterHub
[Service]
User=[username]
ExecStart=/usr/bin/python3 /path/to/tensorboard-routing.py
[Install]
WantedBy=multi-user.targetCode language: Bash (bash)The service can be enabled with the following commands.
sudo systemctl enable tensorboard-routing.service
service tensorboard-routing start
service tensorboard-routing statusCode language: Bash (bash)Singleuser Server Proxy (tensorboard-proxy.js)
To correct incoming paths a proxy needs to be run on the single user server. Copy the following contents into a file named tensorboard-proxy.js.
var http = require('http'),
httpProxy = require('http-proxy');
var proxy6006 = httpProxy.createProxyServer({});
var proxy6007 = httpProxy.createProxyServer({});
var proxy6008 = httpProxy.createProxyServer({});
var server6006 = http.createServer(function(req, res) {
items = req.url.split('/');
req.url = '/' + items.splice(4).join('/');
proxy6006.web(req, res, { target: 'http://localhost:6006' },
function(e) { console.log(e) });
});
var server6007 = http.createServer(function(req, res) {
items = req.url.split('/');
req.url = '/' + items.splice(4).join('/');
proxy6007.web(req, res, { target: 'http://localhost:6007' },
function(e) { console.log(e) });
});
var server6008 = http.createServer(function(req, res) {
items = req.url.split('/');
req.url = '/' + items.splice(4).join('/');
proxy6008.web(req, res, { target: 'http://localhost:6008' },
function(e) { console.log(e) });
});
server6006.listen(6116);
server6007.listen(6117);
server6008.listen(6118);
Code language: JavaScript (javascript)This file will need to be run on the single user server using node. This can be accomplished using a shell script as follows. This will need to be run anytime the single user server restarts.
#!/bin/bash
node /etc/tb/tensorboard-proxy.jsCode language: JavaScript (javascript)Run Tensorboard
After creating a new single user server (Jupyter Lab) open a terminal from the launcher. Run the following the command.
Code language: Bash (bash)tensorboard --logdir=/path --bind_all --port=600X
Use port 6006, 6007, or 6008 for the url paths /user/xxx/tb/, /user/xxx/tb6007/, and /user/xxx/tb6008/ respectively. Don’t leave off the trailing /.
References
- https://github.com/http-party/node-http-proxy