Information about Helm v3 release is stored in a secret within a namespace where release is deployed.
These secrets can be carefully updated to "move" release from one namespace to another.

The following moves Helm release, as visible from output of helm ls -n mynamespace command. It does not move any of the K8s resources which are part of the release.

Using Jenkins Helm chart as an example:

1. reflect on "do I really need to do this?"

2. get the original secret

   kubectl -n default get secret sh.helm.release.v1.jenkins.v1 -o yaml | neat > jenkins_release_orig.yaml
   

3. decode the secret's data.release (it's double-base64 encoded, and gzipped)

   oq -i yaml -r '.data.release' jenkins_release_orig.yaml  | base64 -D | base64 -D | gzip -d > jenkins_release_secret_decoded.txt
   

4. change the namespace of the release, and save the changes into the new file

   NEW_VALUE=$(cat jenkins_release_data_decoded.txt | jq -c '.namespace="jenkins"' | gzip -c | base64 | base64)
   oq -i yaml -o yaml --arg new "$NEW_VALUE" '.data.release = $new' jenkins_release_orig.yaml > jenkins_release_moved.yaml
   

5. verify and apply

   oq -i yaml -r '.data.release' jenkins_release_moved.yaml | base64 -D | base64 -D | gzip -d | jq .namespace
   kubectl -n jenkins apply -f jenkins_release_moved.yaml
   helm -n default ls
   helm -n jenkins ls
   

6. remove old release record (secret) in original namespace

   kubectl -n default delete secret sh.helm.release.v1.jenkins.v1
   

7. rinse and repeat for other versions / deployments of the release

Moving resources to another namespace can be achieved grabbing K8s manifests, modifying them with desired metadata.namespace and redeploying them. It helps to be extra careful with PVs, to avoid loosing them. Whenever possible, consider deleting Helm release, and redeploying it in the new namespace to save yourself a headache.

Like aws-vault is a helper for AWS related CLI tools, k8s-vault is a helper for CLI tools using KUBECONFIG. Unlike aws-vault, "vault" is used as a verb, synonymous to leap, jump, spring, etc..

About two years ago, I've made a script described in Using SSH + Port-Forwarding for K8s CLI tools post.

That CLI script serves as wrapper to other CLI tools using KUBECONFIG. It establishes an SSH Forwarding session via SSH jumphost, while generating temporary KUBECONFIG with server endpoint modified to use the TCP port of the forwarding session.

While the script worked well in my experience, it had to be fixed when newer version of yq introduced incompatible changes. And then again.. The whole dependency on specific version of helper tools is rather annoying..

Recently, I've implemented the same thing in Crystal, learning the language. Enter k8s-vault.cr

It works pretty much the same way, with a slight change in k8s-vault.yaml config format.

The repository's releases page includes statically compiled binary for Linux (x64), as well as dynamic one for macOS.

Tiller runs in-cluster with privileges granted to it by service account, which is suboptimal, since anyone with cluster access would be able to connect to gRCP endpoint Tiller is exposing.

Upcoming release of Helm 3.x solves the issue entirely by removing Tiller, having helm CLI communicate directly with K8s API, thus relying on K8s RBAC for auth/authz. Check out a post A Gentle Farewell to Tiller on Helm blog, and The state of Helm 3 by Thorsten Hans.

While waiting for release of Helm 3, we have to deal with Tiller.. There are many articles discussing Helm 2.x security:

• Exploring the Security of Helm
• Is Your Helm 2 Secure and Scalable?
• Tillerless Helm v2
• and many more

One of the better options is to use Tillerless Helm v2.
Another, is to enable TLS-based auth, issuing SSL certs to every user authorized to connect to a specific Tiller instance (since there can be multiple Tiller instances running, each with it's own set of privileges).

Working with multiple TLS-enabled Helm 2.x clusters is somewhat of a pain, since passing it requires use of environment variables or cli options to select specific certificates, without option to use configuration file.

To make it easier, I've made a wrapper script, naturally. When --tls is passed to the wrapper, it sets necessary environment variables based on passed --kube-context argument, and calls helm binary.

The script also addresses an annoyance of having to match Helm client CLI version with Tiller (server component), by detecting Tiller version, and downloading an appropriate version of helm cli from Helm's Github releases page.

Rancher 'n KIND - running Rancher with ephemeral K8s cluster for local testing.

If you'd like to experiment with Rancher and K8s in local dev environment, KIND is just the right thing for ephemeral K8s clusters. Kind is a tool for running local Kubernetes clusters using Docker container "nodes". It's super helpful when developing K8s, to build images with your changes, and quickly spin up an instance for a test.
Even if you are not developing K8s, kind is a great tool to provision a local K8s cluster for testing.

Fulfilling a need to test Rancher integration with K8s, and run experiments with this setup, I've made a naive helper script to start Rancher instance in Docker, and launch a kind cluster for it to use.

Available as Github Gist: rkind.sh

Having to SSH to jumphost, or (through jumphost) to one of the nodes within the environment is annoying, and slows down cluster-related work, since YAML manifest changes made locally in my text editor of choice are not instantly available on the jumphost. One could use SSH-FS, or another method to continuously synchronize local changes, but there's an easier way. As a one-off it's easy to start an SSH connection to jumphost, with SSH's TCP Port-Forwarding to forward requests to K8s API via localhost port of choice. Scripting the process improves the experience, especially when frequently working with multiple clusters.

Inspired by AWS-Vault tool, I've made utility with similar usage pattern, which automates SSH Port-Forwarding setup for selected (configured) clusters. In this case, "vault" is a verb, synonymous to leap, jump, spring..

UPDATE: k8s-vault has been reimplemented in Crystal (and has no external dependencies, other than SSH client): https://github.com/anapsix/k8s-vault.cr

The whole thing is a BASH script with dependency on jq, yq, grep / ggrep to parse KUBECONFIG (~/.kube/config), nc to check connectivity to K8s API endpoint, and openssh-client to establish connection to SSH jumphost.

Config file looks like this

## k8s-vault
k8s_api_timeout: 5 # in seconds  
ssh_ttl: 10 # in seconds  
ssh_forwarding_port:  
  random: true
  static_port: 32845
clusters: # same as in your KUBECONFIG  
  prod:
    enabled: true
    ssh_jump_host: jumphost.prod.example.com
  qa:
    enabled: true
    ssh_jump_host: jumphost.qa.example.com
  dev:
    enabled: false
    ssh_jump_host: jumphost.dev.example.com

It works by extracting relevant config options from existing KUBECONFIG, and generating new temporary one. Feeding it via environmental variable to instance of whatever CLI tool stated [by k8s-vault]. As long as that tool is capable of using KUBECONFIG environment variable, K8s-Vault can be helpful.

The entire script is available as a GitHub Gist

The script is a POC, and may or may not be reimplemented in Go, Rust, or whatever other language I decide to play with.

UPDATE: It's been reimplemented in Crystal: https://github.com/anapsix/k8s-vault.cr

If job history does not need to be retained, one could use concurrencyPolicy: Replace. However, that will make successfulJobsHistoryLimit and failedJobsHistoryLimit meaningless, as jobs will be replaced each time CronJob schedule kicks off another one.

Perhaps Downward API can be used to get container start time, but I haven't found the right reference for that yet.

I like to be able to see what went wrong in failed job runs. Counterintuitively, using restartPolicy: Never will keep failed pods around, and available to examine.

CronJob with timeout via livenessProbe example

$ ./check_http.sh http://google.com
0,1.659404,200,2,https://www.google.ro/?gws_rd=cr,ssl&dcr=0&ei=QaRUWsWdBqaYjwSBlILoAw  

Add a template and a route to either main or individual rsyslog config (in my case /etc/rsyslog.d/10-docker.conf):

# /etc/rsyslog.d/10-docker.conf
# Log Docker container logs to file per tag
$template DockerLogs, "/var/log/docker_%syslogtag:R,ERE,1,ZERO:.*docker/([^\[]+)--end%.log"
if $programname == 'docker' then -?DockerLogs  
& stop

Restart rsyslog and test your container output:

root@test:~# service rsyslog restart  
root@test:~# docker run -it --rm --log-driver syslog --log-opt syslog-tag=java8u66b17-test1 anapsix/alpine-java java -version  

Observe newly created log file (give it few seconds to get witten to disk):

root@test:~# cat /var/log/docker_java8u66b17-test1.log  
Oct 28 15:50:47 test docker/java8u66b17-test1[2492]: java version "1.8.0_66"#015  
Oct 28 15:50:47 test docker/java8u66b17-test1[2492]: Java(TM) SE Runtime Environment (build 1.8.0_66-b17)#015  
Oct 28 15:50:47 test docker/java8u66b17-test1[2492]: Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)#015  

As of Docker 1.9, tagging logs with --log-opt [syslog-tag/fluentd-tag/gelf-tag] will be replaced with simple --log-opt tag=blah. Using constructs such as {{.Name}}, {{.ID}}, {{.ImageName}} and similar will be possible as well.
For details see GitHub #15384

Reference

Docker Logging documentation: https://docs.docker.com/reference/logging/overview/
Rsyslog templates docs: http://www.rsyslog.com/doc/v8-stable/configuration/templates.html
Rsyslog property replacer docs: http://www.rsyslog.com/doc/v8-stable/configuration/property_replacer.html

Recently, I've noticed that one of the event organizers manually updates ticket count on the site every so often.. sort of a drag. Since our event signup is hosted on Eventbrite, I've looked into API method of accessing the ticket counts and while it's possible, it didn't seem like there is a way of using anonymous OAUTH token to get it and I'd rather not expose a private secret by adding it to a JavaScript request in the site code.

Since Eventbrite does not send free-for-all CORS headers, but instead sends x-content-type-options:nosniff with the response (defeating the jsonp trick) , I could not doing a straight-forward jQuery .get/.ajax. Luckily, there are few free proxies available and it even wouldn't be too difficult to roll my own. Of which I ended up using crossorigin.me as easiest to work with.

Simple enough and no need to update ticket count manually anymore..