Using SSH + Port-Forwarding for K8s CLI tools

by Anastas Dancha

While working with multiple Kubernetes clusters, I came across an annoyance of being unable to reach some of cluster's API endpoints directly from my workstation.

Some production K8s clusters I'm working with have their APIs only available within their respective environments (i.e. production K8s clusters), while others are available directly, via general corp VPN (i.e. non-prod K8s clusters).

Having to SSH to jumphost, or (through jumphost) to one of the nodes within the environment is annoying, and slows down cluster-related work, since YAML manifest changes made locally in my text editor of choice are not instantly available on the jumphost. One could use SSH-FS, or another method to continuously synchronize local changes, but there's an easier way. As a one-off it's easy to start an SSH connection to jumphost, with SSH's TCP Port-Forwarding to forward requests to K8s API via localhost port of choice. Scripting the process improves the experience, especially when frequently working with multiple clusters.

Inspired by AWS-Vault tool, I've made utility with similar usage pattern, which automates SSH Port-Forwarding setup for selected (configured) clusters. In this case, "vault" is a verb, synonymous to leap, jump, spring..

UPDATE: k8s-vault has been reimplemented in Crystal (and has no external dependencies, other than SSH client): https://github.com/anapsix/k8s-vault.cr

The whole thing is a BASH script with dependency on jq, yq, grep / ggrep to parse KUBECONFIG (~/.kube/config), nc to check connectivity to K8s API endpoint, and openssh-client to establish connection to SSH jumphost.

Config file looks like this 

## k8s-vault
k8s_api_timeout: 5 # in seconds  
ssh_ttl: 10 # in seconds  
ssh_forwarding_port:  
  random: true
  static_port: 32845
clusters: # same as in your KUBECONFIG  
  prod:
    enabled: true
    ssh_jump_host: jumphost.prod.example.com
  qa:
    enabled: true
    ssh_jump_host: jumphost.qa.example.com
  dev:
    enabled: false
    ssh_jump_host: jumphost.dev.example.com

It works by extracting relevant config options from existing KUBECONFIG, and generating new temporary one. Feeding it via environmental variable to instance of whatever CLI tool stated [by k8s-vault]. As long as that tool is capable of using KUBECONFIG environment variable, K8s-Vault can be helpful.

The entire script is available as a GitHub Gist

The script is a POC, and may or may not be reimplemented in Go, Rust, or whatever other language I decide to play with.

UPDATE: It's been reimplemented in Crystal: https://github.com/anapsix/k8s-vault.cr