A sane helm chart workflow

Kubernetes  · 

Intro

Helm is a templating and deployment mechanism for kubernetes resources which is endorsed by the kubernetes core team. There are a couple of other possibilities to template kubernetes resources, but using helm charts facilitates finding other developers familiar with the technology.

This blog post describes the helm chart workflow which worked best for us.

You find a repo with a basic helm charts, tests and a simple build file here: https://github.com/ecodia/helm-chart-example

Basic Principles

One repo per chart

While even the official helm charts are managed in one large repo our experience was, that this makes handling change on single charts much more difficult and gives repo editors much more power to destroy stuff than is usually desired.

We wanted to let the developers edit and ship their helm charts themselves, devops style. With having a separate repo per chart, this was easy to coordinate and no team could destroy other teams’ charts by accident.

Helm charts should be treated like any other codebase and the reasons to separate the repos are the same like for all codebases. Decoupling and better control. The additional overhead of managing the release-cycles of all the repos is leveraged by the build automation.

Usage of a chart repository (e.g. chartmuseum)

Just like a docker-registry or maven repository the helm repository allows to easily and reliable manage versioned builds of the helm charts. This allows a clean release process and a clean hand over to the deployment pipeline. Chart repos can be synced between environments (by scripts).

Use docker to run a local chartmuseum for tests or to run it in production. Hardware requirements are minimal.

docker run --rm -it \
  -p 8080:8080 \
  -e DEBUG=1 \
  -e BASIC_AUTH_USER=chartadmin \
  -e BASIC_AUTH_PASS=changeme \
  -e STORAGE=local \
  -e STORAGE_LOCAL_ROOTDIR=/charts \
  -v $(pwd)/charts:/charts \
  --user 1000:1000 \
  chartmuseum/chartmuseum:latest

Setup helm with all plugins

We use the following plugins when working with charts.

helm init --client-only

helm plugin install https://github.com/lrills/helm-unittest
helm plugin install https://github.com/technosophos/helm-template
helm plugin install https://github.com/chartmuseum/helm-push
helm plugin install https://github.com/databus23/helm-diff

Write some tests

Even only having some basic tests which show differences in the resulting yaml structure are a huge improvements against having no tests at all. The tests also allow to test the outcome with different set values, which is especially useful to test “on/off” switches ( `` ).

You find more details on the helm unittest github page: https://github.com/lrills/helm-unittest

And more examples in our demo repo: https://github.com/ecodia/helm-chart-example/tree/master/example-application/tests

A simple example

This simple example checks that the deployment resource exists after rendering the template and that the spec subpath of the yaml file matches the previously taken snapshot. Remember to do a git add tests after running helm unittest the first time, so the snapshots are added to the gitrepo.

When you then run helm unittest . again after some changes to the templates you will see the tests fail and you see the changes in the rendered output. If the changes were made intentionally you can run helm unittest -u to update the snapshots.

Then again remember to commit the snapshots together with your chart changes.

suite: test deployment
templates:
  - deployment.yaml
tests:
  - it: should be of type Deployment
    asserts:
      - isKind:
          of: Deployment
  - it: manifest should match snapshot
    set:
      image:
        repository: test-docker-registry
        tag: 1.2.3-TEST
    asserts:
      - matchSnapshot:
          path: spec

Publish the chart to chartmuseum

Assuming you want to push the chart to chartmuseum, first add your helm repository with the credentials, then push the chart to the registry:

helm repo add --username chartadmin --password changeme localtest http://localhost:8080
helm push . localtest

You can see all available chart versions in the yaml file:

curl --user chartadmin:changeme http://localhost:8080/index.yaml

Use helmfile as an umbrella chart

A helm umbrella chart is a helm chart containing only dependencies to other charts (via the requirements.yaml), so these can deployed as one complete deployment.

roboll/helmfile is a separate tool which allows to deploy multiple different helm charts and also adds a ton of useful features (like templating in the values files).

https://github.com/roboll/helmfile

Automated deployment

Build a jenkins job to deploy the chart to different environments.

You can deploy your charts directly via helm upgrade --install after the chart build and supply environment specific information by different values files which you provide via the -f dev-values.yaml option of the helm command.

You can also use helmfile which is suited very well for this task.

Just create a dropdown jenkins parameter with your environments (e.g. “dev”, “qa”, “test”) which match environments in your helmfile.

Tipps and Tricks